Wang2013 の履歴(No.10)

Question about design.
How does the user run the stream processing engine ?
One approach
- run the stream processing engine on one server.
- It listens to one port to accept command.
- User is a different process, it sends command to the server by socket(register query, register schema).
- Server supports different kinds of wrappers for different kinds of information sources.
another approach
- the user and the engine may be the same process.
- The engine classes are compiled into binary libraries.
- The engine provides a set of classes user can use.
- When the user runs the system, it should provide the wrappers implementing the information source.
- Then just compile some classes is OK, not need to re-compile all of the classes of the engine.

I prefer approach 2 because I think how to get input data and how to deal with output data should be specified by the user. The use is able to get better performance.

try to implement an client-server architecture of the system
- then client can register query and get query result from the server
- so we should think about communication between different PCs
communication mechanism
- the easiest way is to use socket directly, the best way is to use socket is I/O multiplexing or asynchronous I/O
- this is difficult to implement because we should deal with block, buffer and many other things. The best way is to use the method provided by each platform because they can support them by low-level system calls and signals.
  - linux: epoll
  - windows: iocp
- it is also not good to use epoll/iocp/kqueue directly because they are complex and not cross-platform, so there are some libraries to encapsulate them
  - libevent
  - libev
  - boost::asio
- it is not good to use libevent/libev/boost::asio directly because if we use them, we will deal with binary data directly and it is not object-orientd. It is better to use RPC(remote procedure call).
  - RCF( boost::asio)
  - eventrpc(libevent)
  - evproto(libevent)

finish the binary representation of JSON
- now each JSON document has a binary representation other than character stream
finish the memory manager
- memory allocate unit: page (each page is divided into chunks, chunk has a fix size, and one record is saved in one chunk)
finish the queue manager
- void push(Element& element);
- void pop(void);
- void front(Element& element);
- bool isEmpty(void);
- bool isFull(void);
implementing the synopsis
- window synopsis
  - void insertElement(Element& element);
  - void deleteOldestElement(void);
  - void getOldestElement(Element& element);
- lineage synopsis
  - void insertLineage(Lineage& lineage,Element outputElement);
  - void getAndDeleteElement(Lineage lineage, Element& outputElement);
- relation synopsis
  - void insertElement(Element& element);
  - void deleteElement(Element& element);