REPORT/RECOMMENDATION TO THE BOARD OF SUPERVISORS
OF SAN BERNARDINO COUNTY
AND RECORD OF ACTION
September 12, 2023
FROM
LYNN FYHRLUND, Chief Information Officer, Innovation and Technology Department
SUBJECT
Title
Contract with Equinix LLC to Provide Colocation-Based Interconnection Services
End
RECOMMENDATION(S)
Recommendation
1. Approve non-financial Master Country Agreement United States, including non-standard terms, with Equinix LLC for colocation-based interconnection services, effective September 12, 2023, and continuing until the last order expires or is terminated.
2. Authorize the Purchasing Agent to issue a Purchase Order to Equinix LLC for Equinix Order No. SV2 1-229229099005 for colocation-based interconnection services in the total not-to-exceed amount of $250,000, for the period of September 25, 2023, through September 24, 2028, automatically renewing for 12-month terms thereafter, unless terminated or notice of non-renewal is provided by either party.
3. Authorize the Purchasing Agent to issue a Purchase Order to Equinix LLC for Equinix Order No. SV5 1-229232779291 for colocation-based interconnection services in the total not-to-exceed amount of $250,000, for the period of September 25, 2023, through September 24, 2028, automatically renewing for 12-month terms thereafter, unless terminated or notice of non-renewal is provided by either party.
(Presenter: Lynn Fyhrlund, Chief Information Officer, 388-5501)
Body
COUNTY AND CHIEF EXECUTIVE OFFICER GOALS & OBJECTIVES
Improve County Government Operations.
Operate in a Fiscally-Responsible and Business-Like Manner.
FINANCIAL IMPACT
Approval of this item will not result in the use of Discretionary General Funding (Net County Cost). Operating costs are recovered via the Board of Supervisors (Board) approved service rates that are charged to County departments for utilization of the interconnection services provided. The total aggregate cost for the five-year service term on Equinix Order No. SV2 1-229229099005 and No. SV5 1-229232779291, paid monthly, will not exceed $500,000. The colocation-based interconnection services costs are included in the 2023-24 Innovation and Technology Department (ITD) budget and will be included in future recommended budgets.
BACKGROUND INFORMATION
ITD utilizes two Equinix LLC (Equinix) collocation centers to provide a reliable, secure, and redundant direct connection to internet network backbones. The County’s IT infrastructure connects with Equinix’s collocation centers to allow departments to access cloud environments to utilize applications, websites, and data as an extension of the County’s data centers. A component of the overall cloud architecture is the use of a colocation-based interconnection company, which provides a bridge between the County’s datacenters and the multiple cloud service providers in use by County departments. Equinix has a wide network of data centers across the globe to ensure proximity to essential cloud providers like Microsoft Azure and Amazon Web Services (AWS), internet service providers, and content delivery networks which reduces latency and improves overall performance crucial to County business processes. The two requested Equinix Orders are specific to two separate locations, which split the data traffic load and provide redundancy should one of these two servers fail.
On September 25, 2018 (Item No. 20), the Board approved five-year Master Country Agreement United States No. 18-699 with Equinix for the provision of colocation-based interconnection services for the period of September 25, 2018, through September 24, 2023. The Board also approved service orders with Equinix in the amount of $561,400, beginning when services were initially delivered, October 8, 2018, through October 11, 2021. These service orders provided an entry point to multiple cloud service providers via connections to two Equinix data centers.
On April 20, 2021 (Item No. 28), the Board approved Amendment No. 1 to the Equinix Service Order issued under Agreement No. 18-699, increasing the amount by $20,700, from $561,400 to $582,100, for the purchase of additional connections, with no change to the term. The Board also designated the Chief Information Officer (CIO) to execute additional service orders for colocation-based interconnection services, subject to review by County Counsel, provided such documents do not increase the total not-to-exceed amount, or change the total service period.
On September 21, 2021 (Item No. 21), the Board approved Amendment No. 2 to the Equinix Service Orders issued under Agreement No. 18-699 increasing the amount by $500,000, from $582,100 to $1,082,100, for service orders for colocation-based interconnection services, and extended the service term by two years, for a total contract period of October 8, 2018, through September 24, 2023, that automatically renews for one additional month, through October 24, 2023, unless either party delivers a non-renewal notice prior to September 24, 2023. This extension aligned the end date of the service orders and delegation of authority to the CIO with the term of Contract No. 18-699.
The Master Country Agreement United States and attached Global Terms and Conditions (Agreement) is Equinix’s standard commercial contract, which includes terms that differ from the standard County contract and omits certain County standard contract terms. While the parties negotiated certain contract terms to County Standards, Equinix would not agree to all County standard terms. The non-standard and missing terms include the following:
1. The Agreement does not contain the standard contract provisions regarding Government Code section 84308 (Senate Bill 1439) and the disclosure of campaign contributions to a member of the Board or other County elected officer.
• Under the County standard contract the contractor states it has disclosed campaign contributions of more than $250 to any member of the Board or other County elected officer and completed a form providing additional information.
• Potential Impact: The County may have difficulty gathering information about campaign contributions made by Equinix.
2. The Agreement does not require Equinix to indemnify the County, as required by County Policy 11-07, including for intellectual property infringement claims.
• The County standard contract indemnity provision requires the contractor to indemnify, defend, and hold County harmless from third party claims arising out of the acts, errors, or omissions of any person. The standard contract provision for intellectual property indemnity is: Contractor will indemnify, defend, and hold harmless County and its officers, employees, agents and volunteers, from any and all third party claims, costs (including without limitation reasonable attorneys’ fees), and losses for infringement of any United States patent, copyright, trademark, or trade secret (Intellectual Property Rights) by any goods or services.
• Potential Impact: Equinix is not required to defend, indemnify, or hold the County harmless from any claims, including indemnification for claims arising from Equinix’s negligence or intentional acts and intellectual property infringement. If the County is sued for any claim, including intellectual property infringement based on its use of Equinix’s software or services, the County may be solely liable for the costs of defense and damages, which could exceed the total Agreement amount.
3. The County is required to indemnify, defend, and hold harmless Equinix and its Affiliates, owners, officers, directors, employees, and agents against all liability and expenses for third-party claims related to the actions of the County’s authorized persons or associated entities.
• The County standard contract does not include any indemnification or defense by the County of a contractor.
• Potential Impact: By agreeing to indemnify Equinix, the County could be contractually waiving the protection of sovereign immunity. Claims that may otherwise be barred against the County, time limited, or expense limited could be brought against Equinix without such limitations and the County would be responsible to defend and reimburse Equinix for costs, expenses, and damages, which could exceed the total Agreement amount.
4. The Agreement does not include certain standard County insurance requirements, including the waiver of subrogation and naming the County as an additional insured.
• The County standard contract requires contractors to carry appropriate insurance at limits and under conditions determined by the County’s Risk Management Department and County Policy 11-07SP.
• Potential Impact: No waiver of subrogation may allow Equinix’s insurer to sue the County, which could result in expenses that exceed the total Agreement amount.
5. Equinix limits its liability to the County, including attorney’s fees, to the total amounts paid or payable by Customer during the 18-month period preceding the date on which the first claim arises. In addition, Equinix limits its maximum liability for claims relating to the licensed space and the services for a non-recurring charge only or as Smart Hands Services to the amount of the fee for such Services.
• The County standard contract does not include a limitation of liability.
• Potential Impact: Claims could exceed the liability cap and the Agreement amount leaving the County financially liable for the excess.
6. Payment terms are Net 60 with a 0.5% monthly past due charge.
• County standard payment terms are Net 60 days with no interest or late payment penalties.
• Potential Impact: County standard processing time is Net 60 days or more. Failing to pay within 60 days will result in a material breach of the Agreement, which would allow Equinix to terminate the Agreement and seek other legal remedies, including charging the County a late payment charge at a rate of 0.5% monthly, which would exceed the Agreement amount.
7. The term of the Agreement is indefinite, beginning on the date of signature and continuing until the last order expires or is terminated. Orders automatically renew at the end of the initial term for 12-month periods.
• County Policy 11-06SP does not permit indefinite term or automatically renewing contracts unless approved by the Board.
• Potential Impact: There is no end term to the Agreement and the County is indefinitely bound to the terms and conditions of the Agreement until all Orders have expired or been terminated.
8. There is no termination of the Agreement for convenience. Orders may not be terminated during the term and can only be non-renewed by providing 90 days’ prior notice.
• The County standard contract gives the County the right to terminate the contract, for any reason, with a 30-day written notice of termination without any obligation other than to pay amounts for services rendered and expenses reasonably incurred prior to the effective date of termination.
• Potential Impact: The County can only terminate the Agreement and any Order during the term for an uncured breach by Equinix. Any attempted termination by County without cause would result in payment liability for the full Agreement amount, which could result in payment liability where no funds are available due to lack of allocation or loss of funding.
The vendor's business processes have changed, and Equinix is unable to provide a version of the order forms that conforms to the County's standards related to digital signatures. As a result, ITD is presenting unsigned order forms to the Board, which will be executed by the Purchasing Agent concurrent with issuance of the Purchase Orders as detailed in Recommendation Nos. 2 and 3.
ITD recommends approval of the Agreement, including non-standard terms, and the Equinix Orders to continue utilizing Equinix’s colocation-based interconnection services for web server redundancy contracted at their two locations in Santa Clara, and San Jose, California. This ensures a seamless failover as ITD operates various County applications and receives support and professional services from Equinix.
PROCUREMENT
The Board approved Agreement No. 18-699 for colocation-based interconnection services on September 25, 2018, following formal Request for Proposal (RFP) No. ISD118-ADM-2708, released on January 5, 2018. ITD has utilized Equinix since 2018 and has invested heavily in its dynamic and reliable infrastructure for data center services. Switching to a new provider would not be cost effective, as the County would be required to invest in new hardware, software, and hire additional staff to assist in the transition. In addition, a transition to a new colocation-based interconnection service provider would require duplicative costs as it would take a significant amount of time since it could not be completed all at once. There would also be the potential for a significant interruption of services for departments during this process. Over the course of time, ITD has acquired significant technical expertise utilizing Equinix’s portal and is familiar with its processes, procedures, and location. ITD manages the already deployed infrastructure and resolves technical issues with efficacy.
The Purchasing Department concurs with the non-competitive justification for colocation-based interconnection services. The Agreement will be used to accompany future purchase orders issued to Equinix.
REVIEW BY OTHERS
This item has been reviewed by County Counsel (Bonnie Uphold, Supervising Deputy County Counsel, 387-5455) on July 27, 2023; Purchasing (Tevan Stremel, Buyer III, 387-2098) on August 1, 2023; Risk Management (Victor Tordesillas, Director, 386-8623) on July 7, 2023; Finance (Ivan Ramirez, Administrative Analyst, 387-4020) on August 28, 2023; and County Finance and Administration (Paloma Hernandez-Barker, Deputy Executive Officer, 387-5423) on August 28, 2023.