Performance Monitoring Tips

NLINK provides several different ways of communicating with other systems. Depending on how a meta-database is configured and how various External Systems (e.g. SAP, SQL Server or any other system NLINK needs to interact with) are set up, the net throughput of NLINK can be greatly affected. This document provides some guidelines on troubleshooting performance related issues in NLINK deployment.

Introduction

Software can rarely be expected to exist in isolation. In a typical environment, it has to co-exist on with other software from disparate vendors. Given the wide choice of hardware available it makes it difficult to comprehensively indicate how to performance-tune a specific software application. When trying to figure out performance issues basic questions that need to be considered are:

Is hardware causing bottleneck?
Which processes are consuming resources?
Is NLINK overloading the machine?
Is bottleneck other external system (that NLINK is configured to interact with)?
Are there any excessive error (or warnings) generated by NLINK?
Are there any errors/ issues that are logged in either System event log/ Application event logs?

In order to address some or all questions it is required to run performance analysis and examine the hardware on NLINK machine.

Hardware

The hardware configuration that NLINK runs under can affect its performance for better or worse. For details minimum hardware requirements see “NLINK Hardware Requrements”. If your hardware does not meet at least minimum requirements needed by NLINK, you should consider upgrading hardware before addressing any performance issues

Performance monitoring

Microsoft Windows Operating Systems (2008, 2012 R2) comes with tool called “Performance Monitor”. Performance Monitor can be used to monitor the NLINK. See How to Collect Performance Statistics on tips for configuring Performance Monitor. Make sure the sampling rate is set properly so that a long duration of data is captured (over hours or days rathers than minutes or seconds).

Various metrics gathered from Performance Monitor can be used to see where maximum resources and time are being spent and understand bottlenecks in systems.

Available Bytes
A low number (over a long period of time) will indicate that application(s) are starved for memory. Adding more physical memory (RAM) might help performance or moving some applications to another box might also be a factor to consider.
Page Faults / Second
Page faults/ second counter indicates number of hardware page faults, pages which had to be retrieved from hard disk since they were not in working memory. In a typical system there will some low page fault count. Large number of page faults/sec indicates excessive paging.

Virtual Bytes
After certain point of time once all the interfaces reach steady state, the NLINK virtual bytes will fluctuate with in certain range. The period of the NLINK needs to reach steady state will vary by every deployment. Neither the virtual bytes should not increase forever. Practical limit for NLINK virtual bytes 3 GB on 64-bit Windows or 2 GB in 32-bit Windows with /3GB switch or 1.5 GB in 32-bit Windows,
If NLINK's virtual bytes value is consistently increasing, it most probably indicates some sort of memory resource issue, it will need further analysis to determine the cause.
Private Bytes
Similar to virtual bytes, private bytes should reach a steady state value and can fluctuate between certain range. Practical limit for NLINK private bytes is 2.5 GB on 64-bit Windows or 1.5 GB in 32-bit Windows with /3GB switch or 1 GB in 32-bit Windows, If NLINK's private bytes value is consistently increasing, it most probably indicates some sort of memory resource issue, it will need further analysis to determine the cause.
Processor Time
Processor time can provide rough indication of throughput and help identify potential issue. if ”Processor Time” for NLINK is a fraction of the overall time taken to run an Interace, then the other External Systems (e.g. SAP, SQL Server or any other system NLINK needs to interact with) might need monitored to get a better understanding of the throughput issues.
Disk space
Typical NLINK operation will requires 1 GB of free disk space. In most cases NLINK configuration files can take up less than 30 MB. Rest of the disk space requirement depends on the use of advanced features, such as Store and Forward and amount of logging configured in meta-database.
Most run time logs generated can be cleared periodically or backed up to secondary server as needed. NLINK trace file 'NLINK.trc" cannot be deleted or moved when NLINK is running.

Guidelines for running Performance Metrics

Make sure you are running on typical environment as close as possible to that expected in production. The amount of RAM and the number and type of other processes running will make a difference, so ensure that they are as close as possible to those expected in production.
Gather the NLINK statistics over a reasonable period of time to avoid basing your analysis on short-term fluctuations.
Make sure that the data sets being processed by NLINK are as close to those typically expected in production environment.
All external systems (e.g. SAP, SQL server) that NLINK is configured to interact with are running at expected load levels.

Tips and Tricks

If you start to notice a period of sluggish NLINK performance the first thing to do is to use the Windows Task Manager application to determine how much CPU and memory the NLINK is consuming. You should also review how much CPU and memory any other applications are consuming during this period. Often, sluggish NLINK performance can be a result of another application hogging the available system resources.
If there is no apparent lack of system resources and NLINK is still behaving below operational expectations the simplest thing to do may be to stop and then re-start the NLINK. Note that you should do this using the NLINK Management Module (NMM) or from Windows service control manager (SCM). In either case use the Windows Task Manager to ensure that all NLINK processes are actually stopped before restarting NLINK.
If you have re-started NLINK and still seen no improvement in performance the next easiest thing to do is to reboot the hardware that NLINK is running on.
If the NLINK logs folder is take up more and more disk space rapidly then make sure NLINK is not running in debug mode. The debug mode should be only used for short periods of time for troubleshooting issues and should never be enabled during production.
One Interface at a time
NLINK is a multi-threaded application. Multi-threaded applications allow for concurrency, but it is also very challenging to troubleshoot performance issues in multi-threaded applications. The NLINK meta-database can be configured to allow multiple Interfaces to run at same time. Each of the Interfaces runs in its own thread. It is best to isolate the Interface of interest and troubleshoot only that Interface. Make sure (by temporarily modifying the meta-database, if need be) that no other Interfaces are running during the period of monitoring or troubleshooting any one specific interface.

Event log size

NLINK Server uses the Windows Event log to report any issues/problems encountered at run time. Depending on the volume of information being captured, you will need to set the size of Event log accordingly, so that all the events that occur over the monitoring time are capured for analysis. Note that setting Event log to be too big makes it difficult to use. A good guideline would be to set it on the order of 100 MBs (not GBs).

The NLINK Event log in conjunction with “Performance Monitor” can be used to determine acitivity in NLINK.

Log and Trace Files

Interface Log files
Using the NLINK Log Message Action, data specific to an Interface can be written to Interface Log files. These files can contain information relevant to specific Interface. If the meta-database is configured to generate Interface Logs then make sure there is enough information in the messages to capture the key issues.

NLINK Logs and Trace files
NLINK can generate traces of all their activity. These traces along with Event log can provide insight into all the activity in NLINK. Some of the contents of Trace files might be useful only to Junot Systems support.Use the NLINK Management Module (NMM) to activate the NLINK traces. The screen is under the Tools >> Settings menu option. Click on the Advanced Options tab to get to the Activate NLINK Server Trace flag. These trace files are different then those generated using NLINK System Properties Debug Level and Create Dump File.

Any (or all) of these options can generate large volumes of data that can further slow the system down and impose further performance penalties.

These flags should be used with caution and only used to troubleshoot for a limited period of time, and ideally never in a production environment.

NLINK Server running in “debug” mode (with NLINK System Property Debug Level set to Debug) will not reflect a normal production environment.