Previous Page Table of Contents Next Page


SESSION 4
HACCP Evaluation and Risk Assessment


The "care and feeding" of a regulatory HACCP-based system for seafood

Timothy Hansen
United States Food and Drug Administration, USA

ABSTRACT

The United States Food and Drug Administration (US FDA) has almost six years of experience with implementation and management of the Seafood HACCP Regulation. The Office of Seafood has learned a great deal about how to administer a regulatory HACCP system. Our experience can be summarized in three lessons learned.

The first lesson is that seafood HACCP is a high maintenance program that presents certain management challenges e.g. effective communication with stakeholders and within the organization, the need to continually monitor trends and the need to conduct rigorous scientific analysis of regulatory actions and changes that occurred during the implementation phase.

The second lesson is that new management approaches will be needed to effectively address the management challenges presented by regulatory seafood HACCP. The US FDA adopted several new approaches to inspections that include in-depth investigator training, a periodic conference with field investigators, regular planned compliance inspections of foreign importers of seafood and determining a rational basis for inspection frequency. Due to the complexity of the subject matter, US FDA also needed to conduct stakeholder outreach programs in a more effective manner. Some of the innovations that were developed were the Fish and Fishery Product Hazard Guide, uniform HACCP training including periodic updates for both industry and regulators and presentations by Office of Seafood staff who accompany investigators on foreign inspection trips. The Office of Seafood recognizes that the present US FDA web site needs to be re-engineered to be a more effective medium of communication. Improved information management is important in capturing essential information, tracking case assignments and examining past decisions for consistency and uniformity.

The third lesson is that periodic program evaluation is essential to ensuring that a regulatory HACCP system is effective. The Office of Seafood use 23 queries of the HACCP data to track performance in four key areas: domestic processors, molluscan shellfish, importers and foreign processors. The analysis is a planning tool that gives insight as to where scarce resources should be allocated so that food safety can be administered in the most effective way.

INTRODUCTION

The implementation of the Seafood HACCP Regulation began on December 18, 1997 when the Seafood HACCP Regulation 21 Code of Federal Regulation 123 became effective. The US Food and Drug Administration has learned a great deal about the challenges of managing a successful HACCP-based system for seafood. There are three general "lessons learned" from our experience that we think would be helpful to other competent authorities for seafood as they develop, implement and maintain their system for seafood safety.

FIRST LESSON LEARNED: SEAFOOD HACCP IS HIGH MAINTENANCE

Communication of the regulatory expectations of the US Seafood HACCP regulation has been difficult for several reasons. HACCP is a rather complex subject that assumes that there is a base line understanding of college level scientific principles on the part of processing plant management. The managers and employees at the processing level do not always have this level of training, therefore, some remedial science often has to be part of any training program to ensure that everyone understands essential concepts. Also, HACCP is systematic in its approach. Not everyone thinks like a scientist or engineer nor has the training to do so. This often results in confusion about how a plan should be written, implemented or managed.

Moreover, the precise expectations that FDA has developed at the Office of Seafood is not obvious to the regulated industry. The regulatory decisions that are produced are from the collaboration of very high level scientists, policy analysts and processing experts. It is not reasonable to assume that the industry could come to the same conclusion because they would not have the same information or resources. FDA needs to continually communicate through its web-site, field investigators, at conferences and any other means available what we expect from each provision of the regulation.

The communication gap is exacerbated by FDA's limited resources. There are only about 30 consumer safety officers in the Office of Seafood. The Office has had an overwhelming request for training, seminars and participation in international fora that would better inform the international fishing community about HACCP requirements. Unfortunately, we have scarce travel resources and competing priorities that prevent Office of Seafood personnel from participating in many of the training sessions that have been proposed over the years. We do try to attend the most important conferences (such as the International Association of Fish Inspectors [IAFI] World Fish Congress) but cannot attend all of the events.

Communications are also hampered from an organizational standpoint. First, our field investigators do not report to the Office of Seafood. We do not control their field assignments and cannot advise them directly about how to perform a seafood HACCP inspection or think through a HACCP problem. We do generally plan the domestic inspection effort for the year ahead and publish a compliance program that specifies the priorities of seafood inspection but do not oversee the plant inspections. Moreover, the culture of FDA does not always encourage open exchange of ideas that is important for the success of seafood HACCP implementation. All organizations have information filters to some extent and FDA is not unique in this regard.

A HACCP regulatory system will need to be carefully monitored for trends to ensure its success. Key information needs to be captured in order to assess the effectiveness of the program and how to adjust the emphasis so that maximal compliance rates are attained. The FDA has the National Seafood HACCP Database for this purpose (to be discussed below) that tracks every seafood HACCP inspection - domestic, foreign and importer. The data obtained from the database must be analyzed in a meaningful way that tracks progress in key areas. These indices are important to track progress in compliance with all the facets of the Seafood HACCP regulation. Additionally, a database system is prone to errors in entry. A system of quality assurance of data should be instituted to ensure that the data is correct and does not lead to false conclusions.

Since HACCP is based on science it is important that the agency evaluate the scientific basis for all enforcement actions. The Office of Seafood performs in-depth scientific based analyses of every serious HACCP violation. This evaluation ensures that enforcement action that may end up in the legal system is scientifically valid and can be relied upon by the courts. FDA has performed over 1,200 such evaluations since the inception of the Seafood HACCP regulation. A significant percentage of these are not subject to further enforcement action because a flaw was found in the scientific reasoning of the investigator or there is not sufficient evidence to base enforcement charges.

Seafood HACCP represents a paradigm change in food regulation from the traditional sanitation and product sampling schemes of the past. Because HACCP emphasizes prevention rather than the detection of food borne hazards it requires a more proactive approach to management. Changing this approach is difficult for the regulators and the regulated industry. HACCP implementation in the United States started with a creative approach to problems. This was necessary since many of the challenges faced by FDA were never encountered before by the agency. The Office of Seafood went through a creative phase where many new ideas were tried and implemented or abandoned to address particular problems as they arose. FDA's learning curve has reached a plateau and most of the new challenges have been addressed so the creative phase is now being replaced by a maintenance phase. The skills of the creative policy analysts are not as important as management skills to ensure that the Seafood HACCP Regulation is maintained and supported.

SECOND LESSON LEARNED: NEW MANAGEMENT APPROACHES WILL BE NEEDED

The systematic approach to food safety requires new management approaches that may not have been necessary or desirable in the problem detection paradigm of the past. FDA started several new initiatives that were specifically designed to solve some of the challenges presented by a prevention based system. First, the Office of Seafood and the field training staff developed an in-depth inspector certification training. This was a rigorous program that combined extensive classroom work and reading with closely supervised field experience in a real inspection situation. The students were required to demonstrate extensive knowledge of HACCP and food safety as well as FDA HACCP inspection protocols. They must demonstrate their acquired knowledge and abilities through written tests and more importantly in audited inspections. The latter is particularly difficult and the majority of FDA field investigators failed their first attempt. Although this was a difficult certification the results were extremely beneficial to FDA because the investigators had enhanced expertise, followed inspection protocols unerringly and demonstrated a very uniform analysis and follow through for HACCP inspections. It ensures that all firms receive the same scrutiny and treatment by the agency.

Second, as mentioned above, the Office of Seafood does not have direct supervision of field investigators and does not have regular communications with them. Since HACCP principles and regulatory reasoning is rather complex, it is imperative to have some information exchange with the staff that actually do the HACCP inspections. The Office of Seafood started hosting yearly conferences with field investigators. These are usually one week long and include topics on the agenda that are important to both the Office of Seafood and the field staff. It also features frank exchanges about what is working or not working and ideas about how to make changes for improvement of HACCP implementation. The Office of Seafood has gained valuable information about enforcement processes that were not obvious. This has allowed us to correct situations that were barriers to successful implementation of the Seafood HACCP regulation.

Third, the Office of Seafood began organizing 10 to 14 compliance inspections of foreign countries that typically cover 100-120 firms annually. Since both domestic and foreign seafood processors (those without a Memorandum Of Understanding - MOU) must comply to the Seafood HACCP Regulation and since about 77% of seafood consumed in the United States comes from foreign sources, it is important that FDA ensure uniform compliance to all seafood regulatory requirements. Although these inspections only cover a small portion of the 13 000 plus firms that ship seafood to the United States, they are valuable for several reasons. First, the Office of Seafood can assess the sufficiency of regulatory controls by the host country so that appropriate measures can be taken to assure safe seafood for the US consumer. Second, it is a precursor to equivalence or limited compliance MOUs with host countries and is a valuable planning tool. Third, Office of Seafood staff have the opportunity to communicate with the competent authority and the industry that export to the US about seafood HACCP expectations that they will have to meet. Fourth, it is an opportunity to get to know our colleagues in foreign countries and collaborate with them to ensure safe, high quality seafood for everyone. Working with other seafood inspection professionals is both a satisfying experience and important for our collective success in regulating the seafood industry.

Fourth, how often the competent authority conducts an inspection will affect the overall compliance with the regulation. As we all know, firms that have not experienced a regulatory visit will tend to slide in compliance with all necessary requirements. This is human nature, to focus on priorities such as production and fulfilling customer needs.

The Office of Seafood conducted an internal study to determine how often a regulator should conduct inspections that will result in maximizing compliance with the regulations and at the same time conserving precious inspection resources. By using simple regression analysis and comparing the frequency of inspection with compliance to the Seafood HACCP Regulation, we were able to determine that high risk firms ideally needed to be visited twice per year to maintain the level of compliance that we thought acceptable. Medium risk firms were generally not compliant and also ideally needed to be inspected twice yearly. The study showed low risk firms to be more compliant, so inspections of those facilities could be every other year and still maintain the desired level of compliance. Competent authorities implementing seafood HACCP need to think through how they will utilise scarce resources to maximize the compliance to HACCP requirements.

As mentioned above, communication of regulatory expectations is a problem for a new prevention based system such as HACCP. FDA inaugurated several new ways to better communicate and interact with our stakeholders. The most important initiative was the publication of the Fish and Fishery Products Hazard Guide (FFPHG). This is an innovative document that explains how to understand the food safety hazards presented in the firm's operation and how to write and implement a HACCP plan. The FFPHG has been updated as new information, which affects the strategies for optimal implementation of the Seafood HACCP Regulation, became available.

Another management initiative was the collaboration between the industry and the Association of Food and Drug Officials to establish a uniform, inexpensive training program that gives students the information necessary to write and implement HACCP plans in seafood operations. This training has now been extended to Asia and South America and has been quite successful in not only satisfying the regulation requirement for training but communicating essential information to the regulated industry. This training and training for investigators has been updated periodically as we better understand how to manage a HACCP system and get new information about seafood safety.

As mentioned above, another important stakeholder outreach program has been the training and presentations performed by Office of Seafood staff while conducting seafood HACCP compliance inspections in foreign countries. These opportunities have enabled us to communicate directly with foreign competent authority colleagues and the regulated industry.

A user-friendly internet web page is an important tool in disseminating information about regulatory expectations. The Office of Seafood Web site has all the information that our stakeholders need to assist them in complying with all regulatory requirements but it is not presently user friendly. Information is difficult to find. We are now working on an updated design that will hopefully better facilitate information flow to our stakeholders.

The status of regulatory enforcement actions need to be tracked in a large decentralized organization like FDA. The Office of Compliance at the Center for Food Safety and Applied Nutrition has instituted a case tracking system that logs when a case involving serious Seafood HACCP violations is received from the field, who worked on it and what final regulatory decision was made by the Office of Seafood after rigorous scientific evaluation. Without this system regulatory actions may get lost or not processed in a timely manner.

A final aspect of new management approaches to seafood HACCP is the need for improved information management. As previously mentioned the National Seafood HACCP Database was established to capture key information about HACCP inspections under the Seafood HACCP Regulation. The capturing of this information is essential to success of Seafood HACCP management by FDA. Investigators submit forms for each inspection that records the firm's compliance with every provision of the regulation as well as the type of processing and the food safety hazards encountered in the facility. This information is used to perform program evaluation analysis, to be discussed next.

THIRD LESSON LEARNED: PERIODIC PROGRAM EVALUATION IS CRITICAL TO THE SUCCESS OF AGENCY MANAGEMENT OF SEAFOOD HACCP

The Office of Seafood conducts bi-annual program evaluations of key information captured by the National Seafood HACCP Database. This type of activity is required under the Government Performance and Results Act and promised in the preamble to the Seafood HACCP Regulation. Despite these requirements, we would conduct these analyses in any case because they give us critical information about how to manage Seafood HACCP.

We have 23 different queries to the National Seafood HACCP Database that show key information about domestic, molluscan shellfish, importer and foreign processor Seafood HACCP inspections. These indicators show how the program is progressing and where we need to provide resources to effect improvement in compliance in a certain area as well as identifying areas of emphasis in the future. They following are some interesting findings from our analysis of 2001 and 2002 inspections. These examples are intended to show how data can be used to sense problems and promote compliance by the regulated industry.

Figure 1 below shows improvement in the overall rate of compliance for 2001 and 2002. There has been steady improvement in compliance since the inception of the Seafood HACCP Regulation in 1997. The Y axis represents the percentage of overall compliance on the part of domestic firms and shows three groups by year. First are firms that are compliant - "no action indicated" (NAI). Second are firms that have major deficiencies that are willing to correct them - "voluntary action indicated" (VAI). Third are firms that have major deficiencies that are not willing to correct them - "official action indicated" (OAI).

FDA considers NAI and VAI inspections to be in compliance with the regulations. As Figure 1 shows there is significant overall compliance, about 91% in 2002, and that this level has been only incrementally improving. We believe that the 2002 compliance rate represent success of Seafood HACCP implementation in the United States and the agency will be pressing for further improvements in the future.

Another success has been the domestic compliance to control of chemical hazards and parasites. Table 1 below shows substantial compliance and continual improvement of adherence to certain selected Seafood HACCP requirements.

FIGURE 1
Domestic Compliance by Level of Charge and Year. NAI - no action indicated; VAI - voluntary action indicated; OAI - official action indicated.

TABLE 1
Overall compliance rates for selected hazards

Hazard

1998

1999

2000

2001

Aquaculture Drugs

93

97

98

99

Food Additives

89

92

94

95

Physical Hazards

89

94

97

96

Parasites

92

99

99

99

TABLE 2
Domestic US industry success rates by specific seafood HACCP requirement

Mandatory HACCP Program Elements

1998

1999

2000

2001

2002

HACCP Plan Present When Needed

78

78

82

82

85

Adequate Identification of Hazards

66

83

86

86

89

Adequate Identification of CCPs

ND

86

81

84

85

Adequate Identification of Critical Limits

67

66

75

77

80

Adequate Identification of Monitoring Procedures

67

76

78

79

77

Adequate Identification of Corrective Actions

76

83

86

87

91

Met Training Requirements

72

85

86

85

86

Table 2 shows the success of the industry since the inception of the rule in meeting the specific requirements of the Seafood HACCP Regulation. The compliance rates to these requirements needs to be improved, although there has been steady progress since 1998. Adequate identification of critical limits and monitoring procedures lags behind some of the other Seafood HACCP requirements. In the early years of HACCP implementation it was requirements such as failure to identify a hazard or failure to identify a critical control point that lagged. We refer to this as the "red shift". At first firms did not understand how to identify hazards. Once they learned what was expected the next inspection revealed that they did not establish a proper critical control point. The third inspection revealed that they did not identify the right critical limits and monitoring procedures. This shows that HACCP implementation on the firm level is a learning process and that industry benefits from interface with an investigator who by charging them with deficiencies indirectly advises them about Seafood HACCP Regulatory expectations.

Table 3 below shows compliance progress on sanitation requirements of the Seafood HACCP Regulations. Sanitation controls lag behind other HACCP requirements in compliance and need to be improved. Sanitation is an important prerequisite program that supports the HACCP concept and high compliance is essential for ensuring the safety and quality of the processed seafood. Improved sanitation will be a priority in future compliance program directives to the FDA field.

TABLE 3
Domestic US industry success rates by specific sanitation requirement

Mandatory Program Elements

1998

1999

2000

2001

2002

Adequate Implementation of HACCP Monitoring Procedures

72

78

77

77

77

Adequate HACCP Monitoring Records

71

69

68

70

77

Adequate Sanitation Controls

21

34

47

50

51

Adequate Sanitation Monitoring

44

77

76

77

77

TABLE 4
Comparison of US and foreign firms for seafood HACCP success by specific requirement

Mandatory HACCP Program Elements

2001 Domestic

2001 Foreign

HACCP Plan Present When Needed

82

98

Adequate Identification of Hazards

86

55

Adequate Identification of CCPs

84

80

Adequate Identification of Critical Limits

77

61

Adequate Identification of Monitoring Procedures

79

65

Adequate Identification of Corrective Actions

87

94

Met Training Requirements

85

95

Adequate Implementation of Monitoring Procedures

77

72

Adequate Monitoring Records

70

91

Adequate Corrective Actions Taken

91

94

Adequate Corrective Action Records

95

97

Table 4 compares the progress of domestic firms in the United States with overall compliance of foreign processors from data derived by FDA investigators during Seafood HACCP compliance inspections overseas in 2001. The results show that the overall compliance rate for foreign firms was comparable but tended to lag in several areas and showed better progress than US firms in several other areas. This information tells us that FDA should focus on the identification of hazards, critical limits and monitoring procedures during inspections and when advising foreign competent authorities. Foreign firms performed significantly better than their US counterparts in meeting the training requirement and keeping required records.

Figure 2 shows that the control of pathogens in ready-to-eat seafood firms needs significant improvement. A compliance rate of 72% is not adequate to protect public health. This will be an area of emphasis in future compliance programs issued to the FDA field.

Figure 3 below shows US domestic progress on adequate histamine control. It shows that compliance has levelled off at about 83%. Given that histamine is the most common serious food safety hazard in seafood, compliance should be improved to adequately protect consumers from this hazard. The information shows that FDA needs to continue emphasis on histamine prevention.

The above examples are intended to show how data analysis and program evaluation techniques can be used to promote continued improvement of Seafood HACCP regulatory compliance and thus promote food safety and quality through adequate HACCP systems at the processing level. This type of information is useful for planning future inspection effort since it is illustrative of the effectiveness of the Seafood HACCP implementation and how much resources should be applied in the future. Moreover, it helps managers decide where scarce resources should be allocated and therefore promote improvement where it is most needed. Finally this type of information can help maximize food safety and quality oversight by a competent authority by identifying problems, needs and successes.

FIGURE 2
US Domestic Industry Success Rates for Controlling Pathogen Hazards in Ready-to-Eat Seafood


FIGURE 3
US Domestic Success Rate for Controlling Histamine Hazard

CONCLUSION

The implementation of seafood HACCP regulatory regimes requires new management thinking and approaches to address challenges that were not previously posed by traditional inspection systems. The managers of HACCP systems consider the following measures to ensure the success of this complex and promising concept:

Focused and Flexible Management

Competent authorities should consider new approaches and different ways of thinking to address new management problems. There should be adequate resources for Seafood HACCP oversight and management will have to carefully focus on the maintenance of the system.

Good Organizational Communication

Seafood is new and complex, so any implementation will involve a commitment by food safety authorities to communicate difficult and seemingly arcane expectations to the regulated stakeholders. This can be done in a variety of ways such as written guidance, direct intervention, web sites and training.

In-depth Scientific Review of Major Regulatory Actions

The HACCP concept is science-based so regulatory actions taken against the industry should have a solid scientific rationale that shows that the regulatory action is necessary because the inadequacy will result in an unacceptable risk to the consumer.

Creative Approaches to New Problems

The competent authority should be prepared to address management challenges in new ways that will address the unique problems of a preventive system of control.

Periodic In-depth Program Evaluation Will be Necessary

Evaluation of data captured from inspection activity is an invaluable tool to ensure that a Seafood HACCP program is performing adequately in terms of public health. It also helps determine where scarce resources should be allocated through identification of the major problems that need to be addressed as a priority.

The Performance Measurement of HACCP Systems - The Canadian Quality Management Program Strategy

Alfred Bungay, Shelley Ippolito and Rick Grant
Fish Seafood & Production Division, Canadian Food Inspection Agency, Ottawa, Canada

ABSTRACT

The utilization of HACCP systems has been accepted over the past decade as the international standard for the control of food safety hazards. The ability to demonstrate the effectiveness of HACCP-based systems to buyers both nationally and internationally is extremely important to establish and maintain market and consumer confidence. This paper presents a modern strategy to measure the performance of the HACCP-based Quality Management Program (QMP) for the Canadian fish processing industry. The QMP, mandatory in Canada since 1992, first utilized a traditional rating system (Excellent, Good, Satisfactory, Fail) and extensive product sampling. The incorporation of quality system principles into the QMP in 1999 allowed the CFIA to explore the use of performance indicators to support program management as well as internal and external performance reporting. In a recent pilot project, the CFIA tested a performance measurement framework using the QMP: the pilot findings are discussed in this paper. Performance reporting was focused in two areas that are of primary importance to a regulatory agency - "Service Delivery" and "Results". In general, service delivery indicators reflect quantity, quality, timeliness or cost of regulatory services delivered. Two indicators of service delivery were initiated; Compliance Verification Delivery Rate and Enforcement Profile. Results indicators are identified to reflect the food safety contribution from the regulatory activities delivered. Five indicators of results were piloted; Domestic Incident Count, Export Incident Count, Product Compliance Rate (Export), Facility Conformity Rate and Industry Compliance Rate. When examined individually these indicators can provide useful data for trend analysis and decision-making. In combination, performance indicators are a powerful management and reporting solution for a science-based regulatory agency. Together, a performance measurement framework and HACCP can provide a comprehensive and defensible summary to attest to food safety assurance for the Canadian public and international trading partners.

INTRODUCTION

Fish and fishery products are one of the most widely traded food commodities in the world. Canada alone regularly trades fish with more than 120 different countries each year. When HACCP was incorporated into regulatory strategies for fish inspection over a decade ago, it revolutionized the standards, controls and verification processes for food safety hazards. As HACCP became the "passport" for food in the global market economy, regulatory bodies and international committees concerned about safe food have discussed and developed strategies for the recognition of HACCP-based food safety systems between countries to facilitate the trade of safe food.

The goal of this paper is to present work in the area of performance measurement at the Canadian Food Inspection Agency (CFIA) and to advocate further discussion on the usage of performance measures as a basis for recognition of food safety equivalences between nations.

The Quality Management Program (QMP) was the world's first regulatory HACCP-based system, originally developed in the late 1980s and becoming mandatory in 1992. At that time, the QMP mandated specific food safety and quality requirements for the approximately 1200 federally registered fish processing establishments in Canada. The 1992 version of the QMP required processors to develop a documented control system outlining their monitoring, corrective actions and record keeping procedures. The regulatory inspection approach utilized an extensive product sampling scheme and a traditional rating system (Excellent, Good, Satisfactory and Fail) based on the number and level of deficiencies.

As the concepts of HACCP continued to evolve and develop so did the QMP. After several reviews, both internal and external, and in consideration of the impending USA Seafood HACCP legislation, it was decided to re-engineer the program to fully incorporate the HACCP principles. In 1999, amended QMP regulations were passed requiring processors to develop and implement a control system to address all food safety and regulatory requirements. The re-engineered QMP included an audit methodology for regulatory verification, new inspection frequencies, a QMP reference standard and a database designed to support high-level analysis of audit results to advance program management and reporting capacity.

In 2002, the CFIA made significant progress in the area of performance measurement and management. Performance management refers to the use of performance information over a period of time to contribute to the management of an organization. The performance information provides the link between planning and reporting and directs change within the organization. The purpose of performance management is to monitor and improve performance in critical areas by creating accountability to stated goals and objectives. Performance measurement is the process of developing measurable indicators that can be systematically tracked to assess progress being made in achieving predetermined goals.

Performance measurement is well-known globally and science-based departments and agencies in many developed nations are making progress in developing performance frameworks focused on organizational accountability. An application that has yet to be fully explored is the use of performance measurement to support country-to-country arrangements concerning the equivalence of food safety systems. The Canadian government has several such arrangements in place with major trading partners in fish and seafood products. These agreements are structurally sound and sustained with assessment and communication; however, they do not have the rigour associated with specific, measurable, accountable and results-based metrics. It is the opinion of these authors that the addition of performance metrics to HACCP-based equivalence arrangements will contribute to the quantification of achievements in food safety objectives.

QMP PERFORMANCE MEASUREMENT PILOT PROJECT

In 2003, the CFIA conducted pilot projects with the objectives of testing a series of performance indicators and data collection and reporting methods. The QMP was included as one of the pilots. This paper reports on QMP performance information collected over a six month period (1 January - 30 June 2003). The pilot included two performance measurement areas:

Measures of Service Delivery

Service delivery indicators tested in the pilot reflect the quantity of products and/or services delivered against a planned number, historical average, target or program requirement. Service delivery indicators are useful to manage the cost, efficacy and consistency of program delivery resources.

Measures of Results

Results indicators reflect the influence, on a desired outcome, of the program delivery. The QMP pilot was focused on the performance measurement for two key results; Fish meets domestic and trading partner requirements, and Fish processing industry complies with the Regulations.

The following account describes the performance information collected and the analysis that was part of the pilot project.

Indicator 1: Compliance Verification (CV) Delivery Rate (Service Delivery Indicator)

Compliance Verification (CV) refers to the audit based approach that the CFIA uses to verify the effectiveness of industry QMP controls. CV frequencies, as indicated in CFIA policy, are determined for fish processing establishments based on whether the HACCP plan has identified significant hazards or not. CFIA managers plan the number of CVs per month based on the industry profile and operating season. The CV delivery rate is the number of CVs completed divided by the number of CVs planned and represented as a percentage.

Figure 1 shows fluctuation in the CV delivery rate, ranging from 55% to 100% with approximately a 70% average delivery rate over the six months studied. CFIA managers can observe differences in delivery rates with increased focus on smaller geographic units. There will always be external and internal factors which will affect the delivery rate and it is very important to be cognizant of this during data analysis.

Indicator 2: Enforcement Profile (Service Delivery Indicator)

The Enforcement Profile relates the number of written warnings, suspensions and revocations issued in relation to CFIA registrations held by fish processing establishments. Enforcement actions are issued based on information concerning non-compliance to regulations gathered by inspection staff during verification activities.

The QMP has a graduated enforcement approach. Enforcement tools are used in a step-wise fashion as the level of cooperation and ability to maintain effective controls decreases. This indicator of compliance interventions conveys important program delivery information. For example, during normal operations, the expectation would be for a greater number of warnings compared to suspensions and also more suspensions than revocations. The pilot data collected reflects program policy in that the number of warnings exceeded the number of suspensions which in turn exceeded the number of revocations. The data can also be used to demonstrate consistent use of enforcement tools and to identify trends.

The following two indicators (Incident Counts) had few data points (i.e. incidents) identified during the pilot project, which can be viewed as a positive reflection of the QMP but further data collection is required to substantiate trends.

Indicator 3: Domestic Incident Count (Results: Fish meets domestic and trading partner requirements)

The Domestic Incident Count refers to the number of complaints or incidents related to fish products produced in Canada which were traced to a failure of the processor's QMP controls (i.e. the incident would not have occurred if effective controls had been maintained).

FIGURE 1
Compliance verification delivery rate

Indicator 4: Export Incident Count (Results: Fish meets domestic and trading partner requirements)

The Export Incident Count refers to the number of incidents where exported product was found not to meet the importing country's requirements and which should have been prevented through the processor's controls. This is an important indicator as over 80% of the fish products produced in Canada are exported; the United States and the European Union are Canada's principal markets.

Indicator 5: Product Compliance Rate (Results: Fish meets domestic and trading partner requirements)

Exporters to defined markets (e.g. the EU) must obtain a certificate from the competent authority (i.e. CFIA) prior to export. The product lots to be exported are presented to the CFIA for inspection and the rate of compliance to the regulatory requirements can serve as an indicator of the effectiveness of the processor's controls. Data collected during the pilot allowed CFIA to compare product compliance results in the areas of quality and labelling, food safety and foreign country requirements.

Indicator 6: Establishment Conformity Rate (Results: Industry Complies with Regulations)

The Establishment Conformity Rate relates to the number of instances of failure to meet specific areas of requirements in the QMP Reference Standard. The QMP Reference Standard provides guidance for achieving compliance with the Canadian Fish Inspection Regulations. Items are assessed as a "non-conformity", or a "critical non-conformity" when the failure has the potential to impact on food safety. In both cases, the QMP requires the processor to address non-conformities through a corrective action plan determined to be satisfactory to the CFIA.

The Establishment Conformity Rate compares the number of assessments to the number of non-conformities identified. Figure 2 indicates a very high Critical Conformity Rate, over 98%, for food safety requirements. For non-food safety requirements the Conformity Rate averages around 70%. More data and analysis is required to consider if this reflects an actual trend and what the potential impacts may be. Plans are under way to dissect the data to assess individual program areas such as rate sanitation controls, personnel hygiene programs, HACCP controls, and others areas pertinent to QMP. This data will be very useful for trend analysis and to identify priority areas for assessment and improvement. This is an area for future consideration and development.

FIGURE 2
Establishment conformity rate


FIGURE 3
Industry compliance

Indicator 7: Industry Compliance Rate (Results: Industry Complies with Regulations)

The Industry Compliance Rate represents the number of "unacceptable assessments" relative to the number of CVs conducted. The QMP is designed to reflect a cooperative regulatory approach. Where a registered processor is willing and able to address and correct non-conformities they maintain the legal right to hold a registration to process fish. When a processor is unable or unwilling to correct non-conformities, an "unacceptable" assessment is concluded and processing must cease.

Figure 3 shows a very high compliance rate, over 98% for the six months during the pilot project. During January 2003, there appears to have been a lower compliance rate. However, this month represented a small data set and continued data collection is required to permit determination of the significance of variances in the data.

CONCLUSIONS

The QMP pilot performance indicators are valuable for several purposes. Operations managers will utilize the information to make decisions regarding program delivery priorities, while program managers will analyze the information for program policy re-design and continuous improvement. In combination, a set of performance indicators is a powerful reporting tool enabling regulators to tell a complete story on the performance of their regulatory inspection system, both internally and externally.

Regulators of fish processing industries should consider performance measurement as a tool to advance the substantiation of HACCP systems and provide a comprehensive and defensible summary to attest to food safety. HACCP performance information can provide assurances for the public and consumers of fish products in Canada and abroad. Moreover, it can be a significant source of information to share with international trading partners to compare the effectiveness of systems, and facilitating information exchange and the determination of equivalence.

The CFIA will continue to develop and review indicators to address the challenges associated with performance measurement and to share the lessons learned with regulatory colleagues.

FAO/WHO risk assessment for Vibrio spp. in seafoods

I. Karunasagar
University of Agricultural Sciences, Mangalore, India

According to the FAO/WHO definition, risk assessment is the scientific evaluation of known or potential adverse health effects resulting from exposure to foodborne hazards. In the case of Vibrio spp., though pathogenicity has been reported for a number of species (Table 1), serious public health concerns are with respect to V. cholerae, V. parahaemolyticus and V. vulnificus. In this paper risk assessment is illustrated with particular reference to V. cholerae in warm water shrimp that is imported by developed countries in Europe and also Japan and United States of America. The justification for taking up this risk assessment is that several billions of tonnes of warm water shrimp are traded annually and this trade is generally adversely affected whenever there are outbreaks of cholera in shrimp-producing tropical countries. Risk assessment consists of four major steps:

(a) Hazard identification: this involves the identification of known or potential health effects associated with a particular agent

(b) Hazard characterization: this involves qualitative and/or quantitative evaluation of the nature of the adverse effects associated with the biological or chemical agent which may be present in foods. A dose-response assessment should be made if data is available.

(c) Exposure assessment: involves the qualitative and/or quantitative evaluation of the degree of intake that is likely to occur

(d) Risk characterization: this is the final step where integration of hazard identification, hazard characterization and exposure assessment is performed to estimate the adverse effect likely to occur in a given population, including attendant uncertainties.

In the case of cholera and shrimp, hazard identification considered the fact that V. cholerae is a heterogeneous species comprising of more than 200 serotypes. Of these only serotypes O1 and O139 are known to cause cholera. Non-O1/non-O139 serotypes are rarely associated with sporadic cases of gastroenteritis. Therefore the agents involved in cholera need to be clearly identified as choleragenic V. cholerae. Though serology has been traditionally used to characterize V. cholerae isolates, there can be problems due to cross reaction. For example, Shimada et al. (1987) reported that certain environmental non-O1 isolates may agglutinate with polyvalent O1 antiserum. Further, Dalsgaard et al. (2002) noted that V. cholerae O155 agglutinates with commercial O139 antiserum leading to mis-identification.

On the other hand, it is now well accepted that choleragenic V. cholerae produce cholera toxin and this is encoded by the ctx gene. Molecular techniques, like polymerase chain reaction (PCR), can be used to detect the ctx gene (Koch et al. 1993; Karunasagar et al., 1995). This enables specific detection of choleragenic V. cholerae in food and water. Non toxigenic V. cholerae O1 has been isolated from the environment in several studies (Colwell et al., 1981; Sakazaki and Donovan, 1984). With respect to shrimp, Dalsgaard et al. (1995) noted that 2% of cultured shrimp from Thailand was positive for V. cholerae O1. Subsequent molecular studies by the same investigators revealed that these isolates were non-toxigenic (Daalsgaard et al., 1996). This shows the importance of characterization of V. cholerae isolates from seafood when performing risk assessment.

Hazard characterization requires data on the prevalence of the organism in shrimp and the levels of the organism in the food at various stages from farm to fork. In the case of choleragenic V. cholerae, the data on prevalence in wild caught and cultured shrimp is very scanty. A series of studies were conducted in several countries in Asia during the late 1980s under the FAO sponsored shrimp hygiene project. These studies report an absence of choleragenic V. cholerae in warm water shrimp. Thus, from this data it is difficult to predict the distribution of choleragenic V. cholerae in warm water shrimp. On the other hand, frequent testing is done on warm water shrimp at the port of entry in importing countries.

In view of this, the FAO/WHO risk assessment considered this data. Over 20 000 samples were tested in Japan during 1995-2000. Data on 181 samples were available from the US FDA and findings from a survey of 752 samples were available from Denmark. Of the total of 21 857 samples tested only 2 samples imported into Japan from India were positive for choleragenic V. cholerae.

Based on the available data, the prevalence of choleragenic V. cholerae O1 and O139 in exported warm water shrimp is very low at around 0.01%. The dose response curve was generated using human volunteer data for classical and Eltor biotypes and Inaba and Ogawa serotypes of V. cholerae (Cash et al., 1974; Black et al., 1987; Levine et al., 1988).

For exposure assessment, the mean weight per serving of shrimp was estimated by taking a mean of Beta Pert distribution with lowest weight being 25 g, the most probable serving size 150 g and highest serving size 1 000 g. The mean of this distribution was 275 g. The number of servings was estimated based on import data from FAO statistics. The mean risk was expressed as:

where the density (l) is expressed in units of V. cholerae (cfu) per 25 g sample portions.

The expected numbers of cholera cases that may occur associated with any given number of servings is the product of the number of servings and the mean risk per serving. The uncertainty associated with this expected number of cases is characterized by the uncertainty of the dose response (a and b) and the uncertainty of the mean density (l), as inferred from the prevalence data on V. cholerae at the point of entry.

This risk assessment shows that even if all imported shrimp is consumed without any further cooking, the risk of cholera is about 2-4 cases in 100 years. This is an over-estimate because all imported volume has been taken as edible volume and it is common that shrimp are generally consumed after cooking. This would reduce the bacterial numbers by > 6 logs. Thus the risk would be near to zero. This inference is supported from epidemiological data. Cholera is a reportable disease and good surveillance mechanisms exist in most developed countries importing warm water shrimp. This shows that there are no cases of cholera reported due to imported shrimp in these countries.

Shrimp intended for export is a high value item and is handled as per Hazard Analysis Critical Control Point (HACCP) guidelines in the producing countries. Adoption of such care would greatly reduce the risk of contamination of shrimp with choleragenic V. cholerae and the risk assessment further confirms this.

TABLE 1
Vibrio spp, which cause, or are associated with, human infections (after Dalsgaard, 1998)


Occurrence in human clinical specimens*


Intestinal

Non-intestinal

V. cholerae O1 and O139

++++

+

V. cholerae non-O1/non-O139

++

++

V. parahaemolyticus

++++

+

V. fluvialis

++

-

V. furnissii

++

-

V. hollisae

++

-

V. mimicus

++

+

V. metschnikovii

+

+

V. vulnificus**

+

+++

V. alginolyticus

-

++

V. carchariae

-

+

V. cincinnatiensis

-

+

V. damsela

-

+

REFERENCES

Black, R.E., Levine, M.M., Clements, M.L., Young, C.R., Swennerholm, A.M. & Holmgren, J. 1987. Protective efficacy in humans of killed whole-Vibrio oral cholera vaccine and without the B subunit of cholera toxin. Infect. Immun., 55: 116 - 1120.

Cash, R.A., Musci, S.I., Libonati, J.P., Snider, M.J., Wenzel, R.P. & Hornick, R.B. 1974. Response of man to infection with Vibrio cholerae. I. Clinical serologic and bacteriologic response to a known inoculum. J. Infect. Dis., 129: 45 - 52.

Colwell, R.R., Seidler, R., Kaper, J., Joseph, S., Garges S., Lockmn H., Maneval, D., Bradford, H., Roberts, N., Remmers, E., Huq, I & Huq, A. 1981. Occurrence of Vibrio cholerae O-group in Maryland and Louisiana estuaries. Appl. Environ. Microbiol. 41:555-558.

Dalsgaard, A. 1998. The occurrence of human pathogenic Vibrio spp and Salmonella in aquaculture. Int. J. Food Sci. Technol., 33:127-138.

Dalsgaard, A., Huss, H.H., H-Kithikun, A. & Larsen, J.L. 1995. Prevalence of Vibrio cholerae and Salmonella in a major shrimp production area in Thailand. Int. J. Foods Microbiol., 28: 101-113.

Dalsgaard, A., Bjergskov, T., Jeppesen, V.F., Jorgensen, L.B., Echeverria, P. & Daslgaard, I. 1996. Prevalence and characterization of Vibrio cholerae isolated from shrimp products imported into Denmark. J. Food Prot., 59: 694 - 697.

Dalsgaard, A., Mazur, J & Dalsgaard, I. 2002. Misidentification of Vibrio cholerae 0155 isolated from imported shrimp as O serogroup O139 due to cross agglutination with commercial O139 antisera. J. Food Prot., 64(4): 670-672.

Karunasagar, I., Sugumar, G., Karunasagar, I. & Reilly, A. 1995. Rapid detection of Vibrio cholerae contamination of seafood by polymerase chain reaction. Mol. Mar. Biol. Biotechnol., 4: 365-368.

Koch, W.H., Payne, W.L., Wentz, B.A. & Cebula, T.A. 1993. Rapid polymerase chain reaction method for detection of Vibrio cholerae in foods. Appl. Environ. Microbiol., 59: 556-560.

Levine, M.M., Kaper, J.B., Herrington, D., Losonsky, G., Morris, J.G., Clements, M.L., Black, R.E., Tall, B.D. & Hall, R. 1988. Volunteer studies of deletion mutants of Vibrio cholerae O1 prepared by recombinant techniques. Infect. Immun. 56: 161 - 167.

Sakazaki, R. & Danovan, T.J. 1984. Serology and epidemiology of Vibrio cholerae and Vibrio mimicus. Methods in Microbiol., 16: 271 - 289.

Shimada, T., Sakazaki, R. & Oue, M. 1987. A bioserogroup of marine Vibrio possessing somatic antigenic factors in common with Vibrio cholerae O1. J. Appl. Bacteriol., 62: 452-456.

FAO/UNIDO Symposium - Issues and Challenges for Developing and Emerging Economies

Moderators: Amidou Ouaouich (UNIDO) and Hector Lupin (FAO).
Rapporteur: Simon Derrick (UK)

This symposium was organized by FAO and UNIDO to provide participants interested in developing countries issues with the opportunity to express their comments and conclusions about international fish inspection issues. As there was limited time available for this symposium, key participants from each region (Africa, Asia, Latin America and the Caribbean) were asked to prepare short presentations in coordination with other participants from the same region. These are presented in this section.

ASIA AND MIDDLE EAST PRESENTATIONS

On behalf of ASEAN countries:

Presented by: Ms Wee Bee Wah (Malaysia)

Identified problems include:

Indonesian representative

Presented by: Urip Santoso

Vietnamese representative

Presented by: Tam Huynh Le

Oman representative

Presented by: Ismail Mohamed Ali Al-Balushi

AFRICAN PRESENTATIONS

Overall position of African participants on issues specific to developing countries

By: Nancy Gitonga - Kenya; Yvette Dei - FAO Ghana; Luisa Athur - Mozambique

1. Risk Assessment

After listening to presentations and recognizing the paradigm shift towards risk assessment as a preventive control system to assure safety of fishery products, there is general concern on the ability of developing countries to cope.

Risk analysis approaches requires significant data collection before performing risk assessment. Availability of data in developing countries is elusive for many reasons. This situation needs to be addressed if developing countries hope to compete in the global market.

More specifically, there are several specific issues to address:

a) Need to build adequate relevant data for risk analysis and management. In order to do so, there is need for:

- Establishment of adequate accredited and referral laboratories in various regions to generate data.

- Facilitation of regional and international networking to share and disseminate information.

b) Need to build technical capacity to handle risk assessment through training and collaboration.

c) Need to develop a strategy to resolve these issues as developing countries and taking into account our constraints. There is a possible role for FAO, who could act as catalyst to bring parties together to consolidate issues and formulate way forward.

2. Trade Issues

Developing countries are generally more vulnerable to unfair application of trade barriers, especially SPS, than are developed countries. The major reason for this is the inadequate capacity for dispute settlement. Due to this, developing countries accept rejections, bans and other trade embargos with little resistance or input. This is in spite of the fact that over 50% of fish traded in the world comes from developing countries.

In order to sustain the market share for developing countries, which is at risk from unfair non-tariff trade barriers, there is need to identify areas of concern and address them adequately.

Some developments that are emerging could easily be used in unfair trade practices especially for the developing countries. These include:

It is important that developing countries make a deliberate effort to participate effectively in various fora where decisions are made. In this regard, we need our governments, development partners and neutral agencies such as FAO to assist these countries to participate.

Again, there are several specific issues to address:

a) Develop relevant databases that would be used to counter any unfair move to bar trade in the pretext of use of protective measures such as SPS and the precautionary approach, which could be termed as disguised TBTs.

b) Become involved in formulation of standards such as Codex, SPS, etc, so that issues peculiar to developing countries can be taken into account. Such involvement includes:

- Attendance of meetings
- Follow-ups on trade issues that affect developing countries.

c) Build capacity on WTO issues such as TBT and SPS, as they apply to fisheries. This can be achieved through:

- Training/workshops/seminars

- Information and communication technology support to help in gathering and dissemination of information

d) Build capacity in dispute settlement, so that countries can raise issues on areas they feel there is deliberate unfair impediment to trade.

3. Regional Approach to Fish inspection and Quality and Safety

There is need to initiate regional cooperation and collaboration to achieve the following:

4. Equitable Distribution of Revenues from Fisheries

It may be important to sensitize governments on the need to invest in fish safety measures to improve and sustain global markets, through fair distribution of revenues from fisheries. The sensitization process should demonstrate that such measures would contribute to improved economic benefit for developing countries. Failure costs, such as those resulting from fish export bans, have been known to be very expensive.

5. EU Harmonization Process

There appear to be a number of issues with the EU harmonization process which to some extent appear to be unfair trade barriers. Some of the issues include:

a) Poor communication from EU to third countries. On closer examination, there is also poor communication between the Food and Veterinary Office (FVO) Dublin and the European Commission in Brussels. For example, Kenya responded to the FVO Inspector's final report on 26 January 2003, but to date, there has been no communication from DG SANCO. Follow up telephone calls, e-mails and letters have not yielded results. The FVO assures Kenya that they have sent their recommendations to the European Commission in Brussels, but the EC does not appear to have received it though this happened way back in August 2003. The EC needs to set bench marks and time frames for Council Decisions, with regard to harmonization processes.

b) Generally, poor information dissemination from the EU to third countries on new requirements, and even to its own member states, became apparent during plenary discussions.

c) The deadline for harmonization, set as 31 December 2003, need to be extended. This request is based on the fact that the EU ACP/OCT project that has been developed specifically to help those countries that have not been harmonized has not taken off yet and many countries may be disadvantaged.

d) There is need for consistency by the FVO evaluation teams. While diversity from different inspectors is expected and accepted, some of the evaluation teams reports are so varied that sometimes it appears as if the goal posts are being shifted to delay the process.

e) The EC should ensure that their harmonization process is not used by some of their member states to bar trade to third countries, by delaying importing fish from the countries that have not been harmonized. This is because EU member states would be obligated to import fish when a country is harmonized, while they are under no such obligation against bilateral trade arrangements.

f) The African region concurs with the submission from Asian countries that a harmonized rejection procedure from different member states be developed. The methods of analysis, especially microbiological testing, also need to be harmonized. The African region has had experience where one country rejects a consignment and another passes it, though the two lots are from one production batch.

Senegalese representative

Presented by: Mamadou Goudiaby

Ivory Coast representative

Presented by: Yvette Diei

CENTRAL AND SOUTH AMERICAN PRESENTATIONS

Peruvian representative

Presented by: Carlos Algre Salazar

Cuban representative

Presented by: Doris Hernadez Torres

Argentinian representative

Presented by: Ricardo Luis Boeri

Brazilian representative

Presented by: Guilherme A. Da Costa

COMMENTS

Presented by: Olivier Hottlet - Belgian Industry

SUMMARY

Presented by: Grimur Valdimarsson

There is a project underway where the EU will be providing assistance to establish an accredited analytical laboratory in each of 8 African nations and it is expected that FAO could collaborate with this project or be able to establish a similar programme in the near future.

The IAFI is thanked for providing the opportunity to raise and discuss these issues, and it is hoped that in future meetings that developing nations will be more involved in the presentations, although this may require additional preparation time to be provided.

Presented by: Hector Lupin

Equivalence workshop

Moderator: Hector Lupin (FAO).

This workshop was held to discuss the issues relating to equivalence in regulations and procedures across the world. A presentation by Hector Lupin provided a framework for discussion. This proceedings includes written submissions for this workshop.

PRESENTATION TO WORKSHOP BY JOHN EMBERLEY

I would prefer to use the term equivalence rather than harmonisation since harmonisation means exactly the same. This will not ever be possible as there are many ways to achieve the same objectives, all equally valid, but not exactly the same.

Initially after the GATT agreements in 1994 there was a lot of activity as countries established bilateral equivalence agreements. However, this activity has decreased over recent years. A lot of work on establishing the parameters for reaching equivalency was carried out by CCFIECS (Codex Committee for Food Import, Export & Certification Systems) and a number of report documents are available to provide guidance on establishing equivalency agreements.

But why are countries having problems in establishing equivalency agreements?

Things that have been introduced since 1994, such as risk management, have made equivalency much more complex than originally envisioned. Fish have also now been subsumed into the general food agreements and legislation, again making the situation much more complex. There would certainly be benefits to returning to product specific discussions.

Another important issue is that of certification. In developing countries certification takes up to 80% of the available resources of the competent authority. This results in the inspectors not fulfilling other more beneficial activities, such as development and training of the industry. This situation is going to get worse rather than better. Whilst the EU and China have strict certification requirements, other countries, such as the USA and Canada, do not require certification by the competent authority of the exporting country and yet feel that due to the increased cost of production that they are paying for the certification system that they don't need.

The requirement for certification originated in the meat industry and has been applied to fishery products, my personal view is that why is certification required in a system where you have mutual recognition frameworks. The latter can inspire the confidence in the ability of each country to manufacture a safe product.

PRESENTATION TO WORKSHOP BY DR S. SUBASINGHE

I would like to say a few words about the Asian situation and the Asian networks that are working towards harmonisation. In Asia there are several organisations such as:

1. SAR - for South Asian region
2. ASEAN - for the Asia and South East Asian region

Harmonisation of individual systems has started under the auspices of these organisations as a response to problems with exporting to the USA over the last few years. This has focussed on post harvest sectors of the industry and the implementation of HACCP, but has not looked at pre-harvest problems that the region is now facing (especially with respect to shrimp farming).

Buyers expect a unified and integrated approach throughout the region and a better use of resources, since they are limited. There is also a need for investment in training and information is required rather than investment in trade or striving to meet customer expectations.

ASEAN Workshop - Harmonisation of standards for the trade of shrimp

The ASEAN meeting in May 2003 formed a task force which was to present its findings in Bangkok in November 2003. The shrimp industry is the most important sector of the fishing industry in Asia, with the continent accounting for approximately 48% of global trade in shrimp. The socio-economic impact of losing these market shares on the communities that produce shrimp would be devastating and as such governments need to resolve the issue.

Where governments are able to inspect and control the entire industry, as in the tuna industry where there are only a few companies involved, products meet the standards of the international market place. This approach however is not applicable to an industry on the scale of the shrimp industry where there are thousands of individual farms, even within a country.

The industry is fragmented, and thus cannot work together for mutual benefit. This means buyers can play one producer against another. To avoid this we need to work together and strengthen our technical capabilities in order to improve our negotiating power with the major trading organisations.

The ASEAN Workshop in Bangkok will:

I believe that the aims of this workshop are commendable and should be fully endorsed and I would like to thank the organisers of this meeting for giving me the opportunity to speak about them today

AFRICAN PRESENTATION

The detailed African presentation in the FAO/UNIDO workshop covered equivalence issues, and is not repeated in this section.


Previous Page Top of Page Next Page