108 Stations Malaysia

Malaysia, despite its relatively small area, has a substantial variation in its terrain and other factors influencing the airflow dynamics. The hilly and mountainous areas are strongly affected by landslides, while the urban areas are prone to flash floods. Also all areas are subject to diverse weather conditions ranging from severe thunderstorms during the inter-monsoon periods to dry hazy conditions during the South-West monsoon and flooding during the North-East Monsoon.

Upgrading and integrating the various surface meteorological sensor data to provide a comprehensive understanding of the rapidly evolving environment and its impact on human activities, is a necessity for achieving the modernization of The Malaysian Meteorological Department's capabilities for detecting, monitoring and predicting meteorological and hydrological phenomena affecting Malaysia, by implementing the National Integrated Surface Meteorological Observing System - NISMOS. The NISMOS project will address the Malaysian Meteorological Department's primary objective of modernizing and integrating the nations various resources and real-time detection capabilities, and also facilitates the exchange of data at the Local, Regional, and Global levels.

History

In 2003, the Malaysian Meteorological Service (MMS) successfully completed a major system upgrade of the conventional Automatic Weather Station (AWS) system to a locally developed state-of-the-art, futuristic AWS system for the entire nation. The project was called "Nationwide AWS Upgrade". The upgrade involved 38 principal stations, where 22 stations were using existing proprietary analog AWS systems and 16 manual stations.

The AWS system was developed with the following characteristics:
• Intelligent weather sensor or interface that allows monitoring and management up to the sensor level
• Hardware independent i.e. the AWS system shall be able to cater for any brand of weather sensors
• Open source solution
• Source code shall be made available to MMS
• Plug & play system
• End to end web-based solution

Project Overview

MMD currently operates a number of automatic surface weather observing stations, which can primarily be divided into principal stations, climatological stations, agro-meteorological stations and marine port stations. In addition MMD also operates more than 200 manual climatological and rainfall stations.

The Project's primary objective is to integrate the existing AWS and manual observing systems into a single comprehensive automated system using current technologies to provide accurate real-time and archived data for forecasting, now-casting, numerical weather prediction models, climatological and hydrological purposes. The Project intends to build on the successful implementation of the previous AWS upgrading exercise.

The second objective of the project is to automate 108 manual climatological and agro-meteorological stations with real-time communications access.

The system shall then form a single comprehensive integrated automated system using state-of-the art, commercial-off-the-shelf (COTS) cutting edge open architecture technologies and open source software with web-based applications to provide an end-to-end solution that modernizes MMS's capability to detect, monitor and forecast meteorological phenomena and the resulting hydrological impacts, with the added advantage of accurate real-time monitoring over the whole country, and possibly elevate Malaysia to a regional leadership position in weather prediction for the 21st Century. This paper presents a brief overall system overview that includes design concept, its pros and cons, implementation and looks at the possibility to further expand the capability of the system to cater for other sectors.

Project Scope of Work

The scope of work includes the following parameters;
1. Standardize and extend the surface weather-station network by installing 108 unmanned automatic stations.

2. Develop a Network and Instruments Monitoring and Data Analysing System, which will contain the station metadata database for both existing and stations which are to be implemented in this exercise.

3. Provide a Central 1-minute Data Repository with the ability for expansion to accommodate existing inputs and future stations with similar and new inputs.

4. Develop a Quality Control System (QC) which enables both automatic and manual QC measures for real-time and climatological purposes for both existing and future data inputs.

5. Develop a system that is able to both import and export data to other surface-based systems within the Malaysian Meteorological Department.

6. Develop a web-based application visualisation module that is able to provide real-time data streaming and generate products from the acquired data.

Specific Requirements

The system has been designed as a fully functional system to include the following major components:
1. A Sensor suite that measures basic meteorological elements such as temperature, relative humidity, pressure, wind, and precipitation. The sensor suite complies to all the requirements laid down in WMO No.8 (Methods of Observation and Instrument Guide).

2. The acquisition electronics module that collects data from the sensor suite uses intelligent interfaces, processes, performs preliminary quality control procedures and aggregates the collected data, logs the processed data and finally transports the data to the Central Processing Facility in MMD Headquarters using streaming TCP/IP protocols.

3. The sensor sites are equipped with power supplies that comprise either solar-based power supplies or AC based power supplies. The power supplies have self-monitoring capabilities and have the capability for error reporting.

4. The communications interface has the capability of interfacing to any TCP/IP based communications module. Currently GPRS data transmission is used but other faster wireless based communications methods are being explored. GPRS was chosen as the coverage in the country is considerable and the cost is low.

5. The Central Processing Facility is the focal point of the system. It functions as the data collection centre where data processing, aggregation, QC, message generation and dissemination, storage and visualization occurs. The system runs as a cluster and all applications are run as clustered applications with full back-up facilities.

6. The Operating System and all software tools used for the system are all open source implementations. All software applications are web based and comprise of the main ingest module, a data QC and aggregation module which is based on the Nordic model, a maintenance and application module, a storage module with heartbeat monitoring and a visualization module and the communications module which has a VPN to enable data synchronization in the event of communications failure. The real-time streaming 1-sec & 1-min data visualization using a web-based GIS interface ala Google Earth. This component represents the ability of the system to incorporate the latest communications and software technologies, which would enable forecasters to virtually visualize the events as they occur at the remote sites.

Choice of Technology

As a real-time data acquisition, transmission and visualization system the NISMOS poses stringent requirements on both communications, hardware and software and therefore makes extensive use of ICT and modern technologies to integrate all stations to the Central Processing Facility. The technology used needs to be highly durable and accurate, and has to deliver the standards and robustness required of a real-time observing as well as a climatological data aggregation system.

Modular Architecture of NISMOS

To enable easy refurbishment and maintenance of the system in later years, the NISMOS has been designed and built using a modular structure with a set of proper documentation following internationally accepted documentation methodology both on paper and in digital form.

Outcomes

Short Term
• Consolidate and standardize system specifications.
• Improve the use and access to "surface" observations data.

Mid Term
• Reduces user uncertainty associated with data and improves the use and access to "surface" observations for the department and the public.
• Reduces the department's depen dence on proprietary technologies.

Long Term
• A National integrated observing network based on requirements and not distinguishable as either division or programme centric.
• National integrated observing "network of systems" of MMD and non-MMD systems.
• Maximize MMD's contributions to the Global Earth Observing System of Systems (GEOSS).

Pro's & Con's

Pro's:
Low operating & maintenance cost
Less manpower to operate and maintain the system. Most of the observations are automated and the system monitoring can be done centrally. Software problems can be rectified remotely while hardware problems troubleshooting can be done remotely. Once problem identified, technician can be deployed to the site with the necessary parts.

Sharing of real-time data
Raw data, semi processed data and processed data can be selectively requested by the data recipient over the Internet. This data can be sent real-time to the recipient(s) simultaneously. Distribution of various levels of semi processed data allows a wide range other applications to leverage on the data for various usages such as for specific scientific research and etc.

Hardware independent
Almost all AWS systems available in the market today are hardware specific i.e. the sensors, data loggers, data acquisition electronics and communication devices. With the new AWS we allow the usage of universal sensors. We even allow combination of various brands within the entire AWS network. In addition, the set-up will be based on plug & play method.

Open source
The new AWS system was developed based on open source OS (LINUX), RDBMS (Postgress) and programming language (PHP, Perl). As such, there is no licensing and the cost of ownership is low.

Open standard data formats.
Data format based on XML.

Cons:
Higher volume "pipe" required for central data collection.
To handle the increased volume of message traffic at the central data processor more bandwith (a bigger "pipe") may be required.

Implementation

The implementation of the NISMOS project is approaching completion, and should be completed by the first quarter of 2009. Currently all stations have been installed and data transmission using GPRS has been implemented. The Central Processing Facility is also nearing completion, with only the forecast application module yet to be completed. All other application modules are in various stages of testing.

First Results
As stated above data from the stations is being received by the Central Processing Facility. The most interesting innovation has been the data sync module, which automatically senses missing data due to communications breakdown and requests data through a VPN channel from the stations. The other interesting module is the QC module which uses a cascading method of testing data chunks to flag erroneous data. It also allows the QC manager to set rules dynamically. The real-time visualization of data has been another interesting innovation which is now used by forecasters to monitor weather conditions. This module would be an interesting planning tool when made available to the public.

Further Development
As the development of the system continues, one important step would be to integrate the real-time data visualization to a nowcasting and early warning system. Another development would be the use of the QC module to test spatial QC methods.