Skip to Content
Technical Articles
Author's profile photo Andrew Lunde

Escaping EINTEGRITY Hell

If you use NodeJS for anything in your project(and it’s hard not to) you may find yourself stuck in EINTEGRITY hell.  What do I mean?

 

When it happens:

 

If you’re deploying an Multi-Target Application(MTA) to SAP Cloud Platform Cloud Foundry, you may find that it can fail if there is a mismatch in the files your sending as part of your project and the system’s attempt to pull in additional dependent files during the deployment.  This can happen with full deploys where the entire project is zipped up, or partial “push” deploys of single NodeJS modules(Including the app-router, and HDI DB modules which are themselves specialized NodeJS apps).

 

How you know it’s happening:

 

You’ll usually see the module itself fail(doing a push) or the full deploy will retry starting the module 3 times before giving up.  Look at the output and find which module is failing.  The default output will not give you enough information to figure this out.  You need to inspect the logs of the module itself.

cf logs failing-module --recent

Scroll backwards through the log output looking for the message “npm ERR! code EINTEGRITY”.  Here’s some sample output.

   2020-07-06T19:40:39.05-0400 [STG/0] OUT -----> Building dependencies
   2020-07-06T19:40:39.05-0400 [STG/0] OUT        Installing node modules (package.json + package-lock.json)
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! code EINTEGRITY
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! Verification failed while extracting @sap/node-jwt@1.6.11:
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! Verification failed while extracting @sap/node-jwt@1.6.11:
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw== integrity checksum failed when using sha512: 
wanted sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw== 
but got sha512-Wr3ARQWQAWoZqVF3fM/s1z+B5n9Ne+4l/BBncb8OckwYB3P6t3aV0CjaA35dcLo4vLG8aT/Lf+E7c0Lga3e6KQ==. (4939364 bytes)
   2020-07-06T19:41:08.13-0400 [STG/0] OUT npm ERR! A complete log of this run can be found in:
   2020-07-06T19:41:08.13-0400 [STG/0] OUT npm ERR!     /tmp/cache/final/.npm/_logs/2020-07-06T23_41_00_464Z-debug.log
   2020-07-06T19:41:08.14-0400 [STG/0] OUT        **ERROR** Unable to build dependencies: exit status 1
   2020-07-06T19:41:08.43-0400 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 14
   2020-07-06T19:41:08.46-0400 [STG/0] OUT Exit status 223

 

Why it’s happening:

 

When your NodeJS file are being prepared for a deploy, the packaging tool will perform an “npm install –production” for you in each NodeJS modules to create a node_modules folder with all the dependencies(and dependencies dependencies) listed in the package.json file and an accompanying package-lock.json file that is synchronized with it.  All the files are then zipped up into the .mtar file and sent for deployment.

If you run “npm install” within a single module and manually “cf push” it, the effect is similar, just that you did the “npm install” part yourself.

When the module is received by the cloud platform and deployed, the NodeJS buildpack will look at what was sent and look to see if additional dependencies are needed that weren’t sent along.

THIS IS WHERE THE PROBLEM OCCURS!

If it’s determined(and I’m not entirely sure exactly what triggers this) that the buildpack needs to check the dependencies, it will read the package-lock.json file that you sent along and do an integrity check by generating a checksum on the package(the but got checksum) you sent in node_modules OR one that it determined it needed to download from the package repo it wants to pull from and compare it against what was listed in the package-lock.json file(the wanted checksum).  If it finds they don’t match, you’ll get the “npm ERR! code EINTEGRITY” error and the deploy/push will fail.

 

What can you do about it?

 

It seems that the reason this occurs is due to some hiccup that happens when the package is published to the repository.  The tiniest glitch will result in a checksum mismatch.

  1. Wait for the owner of the package to notice the issue and re-publish the package so that the checksum is correct.
  2. If you are pushing the single module manually, you can edit the package-lock.json, find the package and replace the (but got checksum for the wanted checksum) and save the package-lock.json file and re-push the module.

But what if you’re trying to get the whole app initially deployed and the packaging tool keeps re-running “npm install” and putting what it thinks is the proper checksum in the package-lock.json file before whisking it away during the deploy(Case #2).  You could unzip the mtar file, unzip the offending module, fix the package-lock.json file, re-zip it back up and then deploy the mtar file, but what a pain that is.

What if you’re on a deadline(or just want to move forward) and can’t wait for the package to be fixed in the repo(Case #1).  What then?  (Also see the addendum at the end of this post.)

What you CAN do in this situation is to tell npm on your local machine to NOT generate the package-lock.json file. If you look back at the logs, you’ll see this line.

Installing node modules (package.json + package-lock.json)

 

You can do this by running this command.

npm config set package-lock false

Delete the offending package-lock.json file and re-package the mtar file and deploy.

Now this time, the buildpack won’t have the package-lock.json to compare against and the side-effect will be that any dependent packages that it pulls, it will trust that the checksum is correct and allow the installation to continue.

When you do this confirm that it’s working by looking for the following in the logs.

Installing node modules (package.json)

This can get you unstuck when nothing else will.

 

A different kind of hell:

 

I’d be remiss to completely ignore a huge issue with ALL of these approaches.

In keeping with my metaphor…   “Better the devil you know than the one you don’t”.  

Other than Case #1 above, these approaches open up the opportunity for someone to be spoofing the npm repo that’s being used and supplying a poisoned version of the packag with malicious code.

These solutions should be TEMPORARY at best.  It’s YOUR decision to use them or not but at least you know what your options are.  If you do end up turning off package-lock, set a reminder in your calendar to turn it back on and try again.  You may be able to notify the publisher of the package and bring the issue to their attention in order to get the repo fixed sooner, but that may not be enough for your immediate needs.

npm config set package-lock true

I hope this saves someone out there.

 

 

Addendum:

 

After running into this again today, I dug deeper and found another approach that may work in some circumstances.  The trick here is to configure the behavior of the mbt packaging tool so as to NOT pull package dependencies or include them.  For each module that is giving you issues, you can add the following in the module definition.

 - name: module-app
   type: nodejs
   path: app
   build-parameters:
      ignore: ["package-lock.json", "node_modules/"]
      # Suppress the calling of npm install by listing no commands
      builder: custom
      commands: []

This also has the side benefit of making your mtar file a LOT smaller and thus improves upload speeds.  The deployer will have to pull the dependencies during the buildpack step, but hopefully the repo that it’s pulling from will not have the checksum issues described above.

 

 

-Andrew

Partners: If you have a question, click here to ask it in the SAP Community . Be sure to tag it with Partnership and leave your company name in the question so that we can better assist you.

 

Assigned Tags

      7 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Ivan Mirisola
      Ivan Mirisola

      Great blog Andrew Lunde.,

      I think I've encountered this error before. I struggled for a while, but eventually I got to the same conclusion as you.

      However, I believe I've managed it differently (since I didn't quite know what was going on). I disabled the packaging of the nodes_module folder on my mtar. This forced the whole project to download the files again on CF while deploying. But I remember I also deleted the whole node_modules folder locally together with the lock file before building it again.

      Once it deployed than I understood what went wrong.

      Best regards,
      Ivan

       

      Author's profile photo Andrew Lunde
      Andrew Lunde
      Blog Post Author

      Ivan,

      Great point.

      Yes, I've had to do that as well.

       - name: module-name
         type: nodejs
         path: module-path
         build-parameters:
            ignore: ["node_modules/"]
      

      The MBT tools seems to obey these build-parameters and not package the node_modules but the earlier java based tool didn't.

      -Andrew

      Author's profile photo Brian McDonnell
      Brian McDonnell

      Andrew,

      Thanks for this post.

      It seems to be very timely given the recent availability of SAP NPM packages in the default public registry. By that, I mean I think I’ve already hit some of these integrity issues you described when I switched my local development environment to use the public registry.

      The problem isn’t necessarily the local development but more when I attempt to deploy to SAP Cloud Platform Cloud Foundry. What’s happening is that I’ll have everything built successfully locally but when I try to deploy I’ll hit those integrity errors during application staging.

      Is it possible that during app staging on SAP Cloud Platform Cloud Foundry, it is using the old private SAP registry and therefore introducing a high probability of integrity errors given slight variations in the files present in the public npm registry and the private sap registry?

      For example, I’ve been having trouble with is the same @sap/node-jwt module you shared, specifically the most recent version 1.6.11. When I install the app using the private repository I find this in the package-lock.json file:

      "@sap/node-jwt": {
        "version": "1.6.11",
        "resolved": "https://npm.sap.com/@sap/node-jwt/-/node-jwt-1.6.11.tgz",
        "integrity": "sha512-Wr3ARQWQAWoZqVF3fM/s1z+B5n9Ne+4l/BBncb8OckwYB3P6t3aV0CjaA35dcLo4vLG8aT/Lf+E7c0Lga3e6KQ=="
      }

      Yet when I install it using the new public registry it is listed as:

      "@sap/node-jwt": {
        "version": "1.6.11",
        "resolved": "https://registry.npmjs.org/@sap/node-jwt/-/node-jwt-1.6.11.tgz",
        "integrity": "sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw=="
      }

      Note the different values for the resolved and integrity fields.

      For the time being, I’m just using an older version of the module (1.6.6) as a workaround, but I’m curious if you have any insight into how the npm registry config works in CF deployments as I feel that this is very likely the culprit.

      Thanks again,

      Brian

      Author's profile photo Pierre Dominique
      Pierre Dominique

      Hi Brian,

      How did you fix the issue with the node-jwt module? I have the same issue: everything works fine when I deploy from my laptop but it fails when I try to deploy from a GitHub action.

      Andrew Lunde thanks for sharing this, as Brian mentioned this is very helpful with the transition from npm.sap.com to the public npm.

      Thanks,

      Pierre

      Author's profile photo Vladimir Hilderink
      Vladimir Hilderink

      I had same problem, in my case the build stopped reffering to @sap/node-jwt@^1.3.0.

      I solved it pinpointing which direct-module-usage caused me the error by removing the dependencies in package.json half by half. In the end I have found out removing the dependency to @sap/xssec:”2.0.0″ made me able to build the project.

      I upgraded to 3.0.0 and it worked! Then I downgraded again to 2.0.0 and it worked again. 

      Vladimir

      Author's profile photo Vladimir Hilderink
      Vladimir Hilderink

      In an other application I had to update @sap/hana-client from 2.4.202 to 2.5.111 in order to solve. This one I could not downgrade afterward.

      Author's profile photo SAP Seeker
      SAP Seeker

      Thanks Andrew. Below Command helped to solve the issue I faced.

      npm config set package-lock false