Escaping EINTEGRITY Hell

If you use NodeJS for anything in your project(and it’s hard not to) you may find yourself stuck in EINTEGRITY hell.  What do I mean?

When it happens:

If you’re deploying an Multi-Target Application(MTA) to SAP Cloud Platform Cloud Foundry, you may find that it can fail if there is a mismatch in the files your sending as part of your project and the system’s attempt to pull in additional dependent files during the deployment.  This can happen with full deploys where the entire project is zipped up, or partial “push” deploys of single NodeJS modules(Including the app-router, and HDI DB modules which are themselves specialized NodeJS apps).

How you know it’s happening:

You’ll usually see the module itself fail(doing a push) or the full deploy will retry starting the module 3 times before giving up.  Look at the output and find which module is failing.  The default output will not give you enough information to figure this out.  You need to inspect the logs of the module itself.

cf logs failing-module --recent

Scroll backwards through the log output looking for the message “npm ERR! code EINTEGRITY”.  Here’s some sample output.

   2020-07-06T19:40:39.05-0400 [STG/0] OUT -----> Building dependencies
   2020-07-06T19:40:39.05-0400 [STG/0] OUT        Installing node modules (package.json + package-lock.json)
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! code EINTEGRITY
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! Verification failed while extracting @sap/node-jwt@1.6.11:
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! Verification failed while extracting @sap/node-jwt@1.6.11:
   2020-07-06T19:41:00.45-0400 [STG/0] OUT npm ERR! sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw== integrity checksum failed when using sha512: 
wanted sha512-1WWg93DFtt+cZZCuW68LK3PxmyJesWkeMiLzLYvq+eSg0kj4PfUGEBR/oK/4LN2qdMiqtn2/6tpED/vL7REwrw== 
but got sha512-Wr3ARQWQAWoZqVF3fM/s1z+B5n9Ne+4l/BBncb8OckwYB3P6t3aV0CjaA35dcLo4vLG8aT/Lf+E7c0Lga3e6KQ==. (4939364 bytes)
   2020-07-06T19:41:08.13-0400 [STG/0] OUT npm ERR! A complete log of this run can be found in:
   2020-07-06T19:41:08.13-0400 [STG/0] OUT npm ERR!     /tmp/cache/final/.npm/_logs/2020-07-06T23_41_00_464Z-debug.log
   2020-07-06T19:41:08.14-0400 [STG/0] OUT        **ERROR** Unable to build dependencies: exit status 1
   2020-07-06T19:41:08.43-0400 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 14
   2020-07-06T19:41:08.46-0400 [STG/0] OUT Exit status 223

Why it’s happening:

When your NodeJS file are being prepared for a deploy, the packaging tool will perform an “npm install –production” for you in each NodeJS modules to create a node_modules folder with all the dependencies(and dependencies dependencies) listed in the package.json file and an accompanying package-lock.json file that is synchronized with it.  All the files are then zipped up into the .mtar file and sent for deployment.

If you run “npm install” within a single module and manually “cf push” it, the effect is similar, just that you did the “npm install” part yourself.

When the module is received by the cloud platform and deployed, the NodeJS buildpack will look at what was sent and look to see if additional dependencies are needed that weren’t sent along.

THIS IS WHERE THE PROBLEM OCCURS!

If it’s determined(and I’m not entirely sure exactly what triggers this) that the buildpack needs to check the dependencies, it will read the package-lock.json file that you sent along and do an integrity check by generating a checksum on the package(the but got checksum) you sent in node_modules OR one that it determined it needed to download from the package repo it wants to pull from and compare it against what was listed in the package-lock.json file(the wanted checksum).  If it finds they don’t match, you’ll get the “npm ERR! code EINTEGRITY” error and the deploy/push will fail.

What can you do about it?

It seems that the reason this occurs is due to some hiccup that happens when the package is published to the repository.  The tiniest glitch will result in a checksum mismatch.

1. Wait for the owner of the package to notice the issue and re-publish the package so that the checksum is correct.
2. If you are pushing the single module manually, you can edit the package-lock.json, find the package and replace the (but got checksum for the wanted checksum) and save the package-lock.json file and re-push the module.

But what if you’re trying to get the whole app initially deployed and the packaging tool keeps re-running “npm install” and putting what it thinks is the proper checksum in the package-lock.json file before whisking it away during the deploy(Case #2).  You could unzip the mtar file, unzip the offending module, fix the package-lock.json file, re-zip it back up and then deploy the mtar file, but what a pain that is.

What if you’re on a deadline(or just want to move forward) and can’t wait for the package to be fixed in the repo(Case #1).  What then?  (Also see the addendum at the end of this post.)

What you CAN do in this situation is to tell npm on your local machine to NOT generate the package-lock.json file. If you look back at the logs, you’ll see this line.

Installing node modules (package.json + package-lock.json)

You can do this by running this command.

npm config set package-lock false

Delete the offending package-lock.json file and re-package the mtar file and deploy.

Now this time, the buildpack won’t have the package-lock.json to compare against and the side-effect will be that any dependent packages that it pulls, it will trust that the checksum is correct and allow the installation to continue.

When you do this confirm that it’s working by looking for the following in the logs.

Installing node modules (package.json)

This can get you unstuck when nothing else will.

A different kind of hell:

I’d be remiss to completely ignore a huge issue with ALL of these approaches.

In keeping with my metaphor…   “Better the devil you know than the one you don’t”.  

Other than Case #1 above, these approaches open up the opportunity for someone to be spoofing the npm repo that’s being used and supplying a poisoned version of the packag with malicious code.

These solutions should be TEMPORARY at best.  It’s YOUR decision to use them or not but at least you know what your options are.  If you do end up turning off package-lock, set a reminder in your calendar to turn it back on and try again.  You may be able to notify the publisher of the package and bring the issue to their attention in order to get the repo fixed sooner, but that may not be enough for your immediate needs.

npm config set package-lock true

I hope this saves someone out there.

Addendum:

After running into this again today, I dug deeper and found another approach that may work in some circumstances.  The trick here is to configure the behavior of the mbt packaging tool so as to NOT pull package dependencies or include them.  For each module that is giving you issues, you can add the following in the module definition.

 - name: module-app
   type: nodejs
   path: app
   build-parameters:
      ignore: ["package-lock.json", "node_modules/"]
      # Suppress the calling of npm install by listing no commands
      builder: custom
      commands: []

This also has the side benefit of making your mtar file a LOT smaller and thus improves upload speeds.  The deployer will have to pull the dependencies during the buildpack step, but hopefully the repo that it’s pulling from will not have the checksum issues described above.

-Andrew