REPORT/RECOMMENDATION TO THE BOARD OF SUPERVISORS
OF SAN BERNARDINO COUNTY
AND RECORD OF ACTION
July 25, 2023
FROM
LYNN FYHRLUND, Chief Information Officer, Innovation and Technology Department
SUBJECT
Title
Non-Financial Agreements with F5, Inc. for Server Network Traffic Distribution Services
End
RECOMMENDATION(S)
Recommendation
1. Approve the following non-financial agreements, including non-standard terms, with F5, Inc. for server network traffic distribution services that enable the Innovation and Technology Department to provide an uninterrupted, redundant, and secure method of accessing County resources such as websites and computer applications, effective upon electronic acceptance, and continuing until terminated by either party:
a. End User License Agreement
b. End User Service Agreement
c. Program Terms
2. Designate the Chief Information Officer, Assistant Chief Information Officer, or IT Division Chief to electronically accept the agreements in Recommendation No. 1, and future updates to the agreements, subject to review by County Counsel, provided that such updated terms do not substantively modify the terms of the original agreements.
3. Direct the Chief Information Officer, Assistant Chief Information Officer, or IT Division Chief to transmit any non-substantive updated terms to the agreements in Recommendation No. 1 and future updates to the agreements, to the Clerk of the Board of Supervisors within 30 days of acceptance or execution.
(Presenter: Lynn Fyhrlund, Chief Information Officer, 388-5501)
Body
COUNTY AND CHIEF EXECUTIVE OFFICER GOALS & OBJECTIVES
Operate in a Fiscally-Responsible and Business-Like Manner.
FINANCIAL IMPACT
Approval of this item will not result in the use of Discretionary General Funding (Net County Cost). The End-User License Agreement (EULA), the End User Service Agreement (EUSA) and the Program Terms with F5, Inc. (F5) (Agreements) are non-financial in nature and do not commit the County to make any purchases. If future purchases are made under these agreements, the Innovation and Technology Department (ITD) will adhere to County purchasing policies and return to the Board of Supervisors (Board) for approval, if necessary.
BACKGROUND INFORMATION
ITD utilizes F5 web application protection equipment and software to provide County departments and the public with secure and reliable access to County websites, messaging services, and application hosting. F5 equipment and software provides server data capacity services to decrease the burden on servers by distributing network and application traffic across multiple servers. Support and maintenance for this solution is critical to the 24 hours a day, seven days a week operations of ITD’s datacenter, disaster recovery and business continuity efforts. Approving these Agreements will allow ITD to continue to utilize the F5 portal to access software updates, including enhancements to F5 software features and patches for security vulnerabilities.
ITD has utilized F5 equipment, software, support, and maintenance since approximately 2008. On March 29, 2022 (Item No. 30), the Board approved Maintenance Terms and Conditions Agreement No. 22-216, including non-standard terms, with F5 for direct support services that enable ITD to provide an uninterrupted, redundant, and secure method of accessing County resources such as websites and computer applications, for purchase amounts as authorized by County Policy, effective as of March 29, 2022, and continuing until terminated by either party, provided that the Agreement remains in full force and effect with respect to maintenance and support services until the County gives notice of non-renewal of maintenance and support services, and the County ceases all use of the support services. The County is now required to accept Agreements referenced in Recommendations No. 1 to utilize the F5 portal to access software updates.
The F5 EULA, as supplemented by the Program Terms, is F5’s standard commercial agreement, which includes terms that differ or are omitted from the standard County contract. While the parties negotiated certain contract terms to County standards, F5 would not agree to all County standard terms. The non-standard and missing terms include the following:
1. Governing law is New York State.
• The County standard contract requires California governing law.
• Potential Impact: The EULA will be interpreted under New York State law. Any questions, issues or claims arising under this EULA will require the County to hire outside counsel competent to advise on New York law, which may result in fees that exceed the total EULA amount.
2. F5 may assign the EULA without notice or consent in connection with any merger, consolidation, sale of all or substantially all of F5’s assets, internal reorganization or any other similar transaction.
• The County standard contract requires that the County must approve any assignment of the contract.
• Potential Impact: F5 could assign the EULA to a third party or business with which the County is legally prohibited from doing business due to issues of Federal debarment or suspension and conflict of interest, without the County’s knowledge. Should this occur, the County would be out of compliance with the law until it becomes aware of the assignment and terminates the EULA. County Counsel cannot advise on whether and to what extent New York law may permit or restrict a party’s right to assign without an express provision in the EULA.
3. There is no provision in the EULA addressing each party’s responsibility for paying attorneys’ fees.
• The County standard contract requires each party to bear its own costs and attorney fees, regardless of who is the prevailing party.
• Potential Impact: County Counsel cannot advise on, whether and to what extent, New York law may affect a party’s requirement to pay the prevailing party’s attorneys’ fees and costs in a legal action where no specific provision is provided in the EULA.
4. F5’s maximum liability to the County is limited to two times the amount paid for the software in the twelve months prior to the event giving rise to the claim, excluding indemnification, gross negligence, and willful misconduct.
• The County standard contract does not include a limitation of liability.
• Potential Impact: Claims could exceed the liability cap and the EULA amount leaving the County financially liable for the excess. County Counsel cannot advise on, whether and to what extent, New York law may limit or expand the exclusion of limits to the extent prohibited by applicable law.
5. There is no termination for convenience for committed subscription fees.
• The County standard contract gives the County the right to terminate the contract, for any reason, with a 30-day written notice of termination without any obligation other than to pay amounts for services rendered and expenses reasonably incurred prior to the effective date of termination.
• Potential Impact: The County can only terminate the EULA during a subscription period for an uncured breach by F5. Any attempted termination by County without cause would result in payment liability for the full EULA amount, which could result in payment liability where no funds are available due to lack of allocation or loss of funding.
6. Venue for disputes arising under the EULA is in the state and federal courts located in New York, New York.
• The County standard contract requires venue for disputes in Superior Court of California, County of San Bernardino, San Bernardino District.
• Potential Impact: Having a venue in New York, New York may result in additional expenses that exceed the amount of the EULA.
The F5 EUSA is F5’s standard commercial contract, which includes terms that differ or are omitted from the standard County contract. The EUSA is a non-negotiable clickwrap accepted by click to accept. The non-standard and missing terms include the following:
1. Governing law is Washington State.
• The County standard contract requires California governing law.
• Potential Impact: The EUSA will be interpreted under Washington State law. Any questions, issues or claims arising under this contract will require the County to hire outside counsel competent to advise on Washington law, which may result in fees that exceed the total EUSA amount.
2. All disputes arising under the EUSA, except for intellectual property issues, must be settled by binding arbitration.
• The County standard contract does not require arbitration.
• Potential Impact: Arbitration decisions are not appealable. In addition, disputes that might otherwise be settled in small claims court would incur arbitration costs that would exceed the costs of a small claims action, and the EUSA amount. County Counsel cannot advise on whether and to what extent such arbitration provisions may be enforceable against a government entity under Washington law.
3. F5 may assign the EUSA without notice or consent at any time.
• The County standard contract requires that the County must approve any assignment of the contract.
• Potential Impact: F5 could assign the EUSA to a third party or business with which the County is legally prohibited from doing business due to issues of Federal debarment or suspension and conflict of interest, without the County’s knowledge. Should this occur, the County would be out of compliance with the law until it becomes aware of the assignment and terminates the EUSA. County Counsel cannot advise on whether and to what extent Washington law may permit or restrict a party’s right to assign without an express provision in the EUSA.
4. The prevailing party in an arbitration is entitled to recover all reasonable costs and fees, including attorneys’ fees.
• The County standard contract requires each party to bear its own costs and attorney fees, regardless of who is the prevailing party.
• Potential Impact: If either party institutes any legal proceedings related to the EUSA, the prevailing party will be entitled to recover reasonable costs and fees, which could exceed the total EUSA amount.
5. The County is required to indemnify F5 against third party claims related to: (a) the County’s breach of the EUSA, any license applicable to third party software or third party applications, or the service policies; (b) the County’s use, disclosure or other handling of New Data (as that term is defined in the Service-Specific Terms) in violation of the EUSA; (c) the County’s violation of any applicable law, rule or regulation; (d) an allegation of infringement or misappropriation of any third party’s rights, in connection with F5’s use of the County Data under the EUSA; (e) the County’s deployment and/or configuration of any County Application; or (f) the County’s gross negligence, fraudulent misrepresentation or willful misconduct.
• The County standard contract does not include any indemnification or defense by the County of a contractor.
• Potential Impact: By agreeing to indemnify F5, the County could be contractually waiving the protection of sovereign immunity. Claims that may otherwise be barred against the County, time limited, or expense limited could be brought against F5 without such limitations and the County would be responsible to defend and reimburse F5 for costs, expenses, and damages, which could exceed the total EUSA amount. County Counsel cannot advise on, whether and to what extent, Washington law may limit or expand this EUSA term.
6. The EUSA does not require F5 to meet the County’s insurance standards as required pursuant to County Policy 11-07.
• County Policy requires contractors to carry appropriate insurance at limits and under conditions determined by the County’s Risk Management Department and set forth in the County standard contract.
• Potential Impact: The County has no assurance that F5 will be financially responsible for claims that may arise under the EUSA, which could result in expenses to the County that exceed the total EUSA amount.
7. F5’s maximum liability to the County is limited to the amount paid for the services that gave rise to the claim during the twelve months prior to the incident giving rise to the claim and only for the County’s inability to use the services or any unauthorized access to, alteration of, or the deletion, destruction, damage, loss, or failure to store any County data.
• The County standard contract does not include a limitation of liability.
• Potential Impact: Claims could exceed the liability cap and the EUSA amount leaving the County financially liable for the excess. County Counsel cannot advise on, whether and to what extent, Washington law may limit or expand the exclusion of limits to the extent prohibited by applicable law.
8. Payment terms are Net 30 date of invoice with late payment penalty interest of 1.5% per month, plus all bank charges and costs of collection, including attorneys’ fees. In addition, F5 may terminate or suspend services.
• County standard payment terms are Net 60 days with no interest or late payment penalties.
• Potential Impact: The County standard processing time is Net 60 days or more. Failing to pay within 30 days from the invoice date will result in a material breach of the EUSA, which would allow F5 to terminate the EUSA and seek other legal remedies, including charging the County interest at a rate of 1.5% per month, bank charges and costs of collection, which would exceed the EUSA amount.
9. The term of the EUSA is indefinite beginning on the date that the County accepts the terms, or registers for an account, if earlier and continuing until termination by either party.
• The County standard contract gives the County the right to terminate the contract, for any reason, with a 30-day written notice of termination without any obligation other than to pay amounts for services rendered and expenses reasonably incurred prior to the effective date of termination.
• Potential Impact: There is no end term to the EUSA, and the County is indefinitely bound to the terms and conditions of the EUSA until termination by either party.
10. There is no termination for convenience.
• The County standard contract gives the County the right to terminate the contract, for any reason, with a 30-day written notice of termination without any obligation other than to pay amounts for services rendered and expenses reasonably incurred prior to the effective date of termination.
• Potential Impact: The County can only terminate the EUSA for an uncured breach by F5. Any attempted termination by County without cause would result in payment liability for the full EUSA amount, which could result in payment liability where no funds are available due to lack of allocation or loss of funding.
11. Venue for arbitration under the EUSA is in Seattle, Washington.
• The County standard contract requires venue for disputes in Superior Court of California, County of San Bernardino, San Bernardino District.
• Potential Impact: Having a venue in Seattle, Washington may result in additional expenses that exceed the amount of the EUSA.
ITD recommends approval of the Agreements, including the non-standard terms, to allow ITD to continue to utilize the F5 portal for software updates including future updates and enhancements to F5 software features and patches for security vulnerabilities.
PROCUREMENT
The Agreements, including non-standard terms, will be used to accompany future purchases to be approved, as necessary, pursuant to County Policy 11-04 Procurement of Goods, Supplies, Equipment and Services, provided that F5 does not substantively modify the terms of the Agreements.
In 2008, F5 was purchased and implemented as the server network traffic distributor for websites and applications for the County. ITD is currently reviewing comparable solutions and will conduct a formal Request for Proposal to obtain a replacement product no later than July 2024.
REVIEW BY OTHERS
This item has been reviewed by County Counsel (Bonnie Uphold, Supervising Deputy County Counsel, 387-5455) on June 12, 2023; Purchasing (Tevan Stremel, Buyer III, 387-2098) on June 27, 2023; Risk Management (Victor Tordesillas, Director, 386-8623) on June 7 2023; Finance (Ivan Ramirez, Administrative Analyst, 387-4020) on July 3, 2023; and County Finance and Administration (Paloma Hernandez-Barker, Deputy Executive Officer, 387-5423) on July 7, 2023.