How Remote Testing Infrastructure Can Streamline Network Pen Testing

There are two types of solutions that we chose discovered to aid in our need for a robust remote testing infrastructure, (1) Onsite Network Implant (ONI) and (2) Firebase. Both solutions present unique advantages and disadvantages based on their usage and implementation.

Onsite Network Implant (ONI)

NCC Group has used this solution since 2016 and is an evolution of Red Team "drop boxes" used for short-term deployments for surreptitious access. This solution derives from a small form factor computer running a Unix-like operating system, typically Kali Linux. To avoid complex firewall configuration changes, the computer logs into a cloud-based host via SSH to establish an encrypted tunnel.

NCC Group consultants can log in to the server and control the ONI device through the tunnel. This allows consultants to have a physical presence on the network without exposing internal systems to the internet. It is also worth pointing out that ONI is platform-agnostic and can support any cloud provider that is convenient for the client.

Much of this functionality is automated so that the onsite personnel need only provide power and network connectivity. If DHCP is not available, we can preconfigure a static IP address. Regardless, NCC Group works closely with the client to ensure that ONI has access to the appropriate networks that are in scope. In the case there are many security boundaries, multiple systems can be deployed. Multiple ONI deployments are also useful for wireless pen testing.

This use case isn't ideal and may be irrelevant now that the lockdown has abated. Still, it's serviceable for clients who need a quick turnaround, tight budgets, or to review a deployment rolled out by a managed service provider.

Advantages:

• Requires minimal support from the client; custom-built for each deployment to be as frictionless as possible.
• Hardware-agnostic and can be deployed on hardware ranging from compact desktops to "stick" PCs.

Disadvantages:

• Virtualized solutions are available, but they require effort on the part of the client.
• Fewer granular controls for the client to control access locally, although end-user accounts exist should troubleshooting become necessary.
• Requires one support day for every two-week engagement, so expensive to run.
• Single point of failure by an Individual rather than a corporately managed solution, including all associated risks.
• Each engagement incurs the cost of running an AWS host (possible complexity of billing that per job?)

Firebase

Firebase is the chosen solution for delivering remote security consultancy services in the UK, Europe, and APAC, and it has a strong and growing presence within North America. In total, Firebase has enabled NCC Group to deliver over 27,000 days of consultancy.

The Firebase project commenced in 2018 to build a solution based on best security practices, which could be offered to customers at no cost.

Firebase can be built and issued to customers in 30 minutes, requiring no more than 5 minutes of a person's time to do so. Firebase officially launched in March 2020, prior to the pandemic. 

Firebase is offered to customers in two versions, as a virtual machine or physical appliance. Both have exactly the same functionality. The virtual appliance is supported supports on VMware ESXi and Microsoft Hyper-V, and there are supported cloud versions for AWS, Azure, Google Cloud, Nutanix, and Oracle Cloud.

A virtual Firebase appliance can be offered to customers at no cost, and customers can have as many as they need to fulfill their network penetration testing requirements. Physical Firebase appliances get billed at the cost of the hardware, which is minimal as that cost gets based on a small form factor appliance, and each region has the hardware and supporting infrastructure in place now to deploy either.

As Firebase can be deployed in 30 minutes at no cost, it is perfect for quick jobs where you need a disposable appliance and is equally suitable for long-term projects. Multiple customers that have tried Firebase gained confidence that it works, recognized the cost savings from simplifying the setup of jobs and travel and subsistence savings, and have since moved all of their testing remotely via Firebase.  

Firebase is a fully managed appliance that runs on a hardened Ubuntu operating system and uses Docker containers. Regardless of the type of work, NCC Group can quickly upload a container with the tools consultants need. For example, there are traditional containers for pen testing, managed vulnerability scanning services, incident response, and security improvement and remediation, amongst others. There is a tried and tested formal documented process for the delivery of PCI engagements as agreed by NCC Group's QSAs. 

Deploying Firebase is simple. The customer plugs in a pre-built physical appliance or downloads and deploys a virtual Firebase. The image is under 5GB, so it takes less than 5 minutes to download and is unique to that customer. When the customer wants NCC Group to test, the customer logs into an interface and clicks start. Firebase establishes an outbound connection over TCP/443 (HTTPS) to NCC Group's endpoint in each region. Firebase is proxy compliant, supporting both HTTP and SOCKS with authentication. All data resides within NCC Group's on-premise environment, so no customer data is stored in the cloud to satisfy those clients who must ensure their data does not leave their region.

The Firebase solution has been built with security best practices in mind, as it represents NCC Group's capability and must not expose our clients to risk. An extensive architecture document details the solution to provide clients with absolute confidence. Internally, the authentication process for consultants to access a client Firebase is robust and fed from the scheduling system, thereby ensuring that only the named consultants, as agreed with the customer, may access the customer environment.

The consultant assigned to the job must connect to a Firebase consultant VPN by supplying a valid certificate plus credentials + One Time Pin. However, the consultant can only link to the customer Firebase once the client has started the connection, as it is an outbound connection, ensuring the customer is always in control. The global Firebase support team may revoke or control access granularly from a central point.

Annually the Firebase appliance build and supporting infrastructure are pen tested, and a customer-facing report gets produced. Security patches are automatically applied four times a day, and logs get extracted from each customer Firebase, including commands run on the base operating system. SNMP monitors each appliance's health and has anti-virus and file integrity monitoring software installed.

The authentication model design is such that Firebase cannot be connected from the client network, thereby reducing the risk exposure, and instead can only connect once the VPN has been established. Consultants connect with named accounts rather than a shared root account, so we know who connected, when, and what they did in accordance with security best practices.

A regional Firebase support team is located in the UK, Europe, APAC, and North America.                                                

Advantages:

• Established enterprise-level product that is the default remote consultancy delivery tool in the UK, Europe, and APAC, with a growing presence in North America
• Can be built and issued to a customer in 30 minutes via an automated build. Requires no more than 5 minutes of an NCC Group employee's time to do so. The customer can deploy and test it in 10 minutes, so it's perfect for quick or long-term engagements.
• Can deploy as either a virtual machine or a physical appliance. The virtual appliance supports all major hypervisors and cloud platforms.
• Firebase can be at no cost to the customer (as a virtual machine), and the physical appliance is available at cost (minimal as it's a small form factor appliance).
• Enables customers to request security testing at short notice with minimal customer setup time required.
• Firebase uses Docker containers, so the customer can quickly add, no matter the job, a new container or new tools as different teams deliver services for customers.
• It connects outbound to our VPN endpoint via TCP/443 (HTTPS) and is proxy aware, thereby minimizing any changes the customer may need to make to their environment.
• Built with security best practices in mind from the ground up. Supported by corporate IT. Granular access control with named accounts and full accountability/audit control.
• The customer can revert each container, so no test data resides on Firebase at the end of an engagement.
• Fully managed appliance that receives security patches four times per day.
• Pen test annually with a customer-facing report available.
• Supported by regional support teams to provide near 24x7 support. Suitable for protectively marked environments.

Disadvantages:

• No remote testing capability can replace the value of collaborating, in real-time and onsite, with the client. Remote capabilities will continue to be an important aspect of delivery, but they will not likely completely replace working side-by-side with them.

Although NCC Group already had remote infrastructure assessment capabilities, pandemic and economic pressure incentivized the firm to advance further and market those services.

This resulted in cost benefits for all organizations involved and has allowed work to continue in circumstances where it may not have otherwise. ONI and Firebase are not a panacea, however. In scoping and planning the instrumentation for any engagement, NCC Group must carefully consider what is fit-for-purpose, fit-for-risk, and what will provide the most value and impact for the client.

By continuing to enhance these remote capabilities, NCC Group will remain able to offer our services in an ever-changing threat landscape, up to and including a global pandemic.