[SIT Belgium 2019 Recap] Debugging Node.js Applications in SCP CF. Part 2: Remote Debugging in Production Environment

Intro

This is a recap of the session “Troubleshooting Node.js applications: Debugging in SAP Cloud Platform Cloud Foundry” that took place in a recent SAP Inside Track Belgium 2019 event.

Slide deck of the session is available at SlideShare.

Overview of the session recap blog series:

Production environment constraints

In the previous blog of this series, we got familiar with steps that are required to prepare an environment for debugging a Node.js application that is deployed to SCP CF and how to conduct debugging. One of essential steps was to start the deployed Node.js application with an enabled Inspector, so that we can later attach a debugger to an Inspector port through SSH tunnel.

What if starting the Node.js application with an enabled Inspector is not an option? For example, the application has been deployed to a production tenant – and running applications with an enabled debugger port right at application startup is not really a very good idea for production environments.

What if we acknowledge and respect this constraint, but are not able to reproduce an issue in other, non-production environments and need to debug the application in the only affected environment – production?

There is a technical possibility to overcome these obstacles and still perform application debugging in such an environment. Before we proceed to description of this technique, I would like to encourage everyone who is tempted to apply it in practice, to carefully assess risks (such as security, performance, consistency, etc.) and potential impact on the debugged application and environment where it runs, and evaluate consequences associated with running debugging session in production environment. It is strongly recommended to apply the below described technique with high caution.

Enable Inspector

An Inspector can be enabled in an already running Node.js application by sending a specific signal – USR1 – to a process of the application. To achieve this, we shall enable SSH access to an application container as described in the previous blog and establish SSH session to the application container. After this is done and we are logged into the application container over SSH, it is firstly required to identify process ID of the Node.js application – for example, using a Linux command ps -aux.

Before we proceed to enabling an Inspector, we can verify and confirm that an Inspector does not run yet by listing ports that are listened by the Node.js application with the help of a Linux command netstat -anp | grep {PID} – at first, the only port that the application is listening on, is an HTTP port (8080).

Next, the corresponding signal can be sent to the application using a Linux command kill -s SIGUSR1 {PID}.

Now, let us check which ports are listened by the application again – as it will be seen this time, the application now not only listens on an HTTP port, but also listens on the port 9229, which is an Inspector port.

Debugging

This part does not differ from what has already been covered in the previous blog, as we now have a Node.js application that runs in SCP CF with enabled Inspector and SSH access is enabled to the application container, so port forwarding for an Inspector port of the Node.js application can be set up through SSH tunnel, and earlier described techniques of remote debugging can be utilized.

Disable Inspector

After we are done with debugging, we would normally disconnect (detach) a debugger and implicitly close debugging session. But wait – if we do this right now, we will not leave the application in its original state here, as when the application has been deployed and started, an Inspector has not been started initially, and now although debugging session is closed, an Inspector remains enabled and listening for incoming connections. We shall address this and ensure we perform necessary cleanup activities and disable an Inspector.

To enable an Inspector, we could benefit from using a specific signal and sending it to a process of the application – in contrast, this approach cannot be used to disable an Inspector as currently Node.js runtime does not provide out of the box capabilities to disable an Inspector from outside the process.

One of options is to restart the Node.js application – given originally it was started with a disabled Inspector, restart of the application will result in starting it with a disabled Inspector again. Although this is technically feasible, the option does not look very optimal, as it requires application restart, and it is something we would rather avoid in a production environment.

Alternative options are as following: in absence of standard mechanisms to disable an Inspector from outside the process, we either need to disable it from within the process using internal API, or introduce a custom mechanism that will trigger internal API to disable an Inspector and that will be accessible from outside the process.

An option that we will focus our attention on in this blog, is to disable an Inspector from within the process by direct invocation of corresponding internal API using Node REPL console. While debugger session is still connected to the application, in Node REPL console, we can make use of Inspector API and trigger deactivation (closure) of an Inspector by calling its method require('inspector').close():

This will instruct Node.js runtime to disable an Inspector and shut down listener on the corresponding port that was in use by an Inspector. This can be verified by listing ports that are listened by the Node.js application one more time and observing that the application is listening on an HTTP port only, suggesting that an Inspector has been disabled.

Another – more advanced – option originates from the approach described above, but does not require invocation of Inspector’s method close() in Node REPL console. Instead, the application can be enhanced by implementing and registering signal event listener for a dedicated process signal that is not in use yet: process.on({SIGNAL}, handler). The handler shall implement similar logic as mentioned above and disable an Inspector if it is enabled. With this enhancement in place, an Inspector can be disabled by sending a corresponding signal to a process of an application using an approach like the one that was used earlier to enable an Inspector in a running Node.js application.

Together with certain advantages (an Inspector can be disabled by sending signal to a process from SSH terminal / by external tools), the latter approach has a certain disadvantage – necessity of adding additional code that shall handle a newly introduced signal and disable an Inspector when such code is received by the application. Although corresponding handler implementation is not that complex, it is going to become a part of a code base of the application and is likely to become repeated boilerplate code, if this approach is expanded and rolled out to multiple applications as a pattern.

Assuming remote debugging of Node.js applications in production environment shall be done by exception in a minimal number of circumstances, and considering advantages and disadvantages of both described methods, I personally tend to prefer the method that implies manual invocation of Inspector’s method close() in Node REPL console at the end of debugging session.

When an Inspector is disabled and given the Node.js application remains up and running, as a final step, we can also disable SSH access to the application container using a CF CLI command cf disable-ssh that was mentioned in the earlier blog.