Node CSV version 0.1 and future developments
By David WORMS
Jul 21, 2012
Never miss our publications about Open Source, big data and distributed systems, low frequency of one email every two months.
The Node CSV parser has just reach version 0.1 which close the 0.0.x releases. Started almost 2 years ago, the project has received a tremendous amount of participation in the form of bug reports, pull request and emails. It is used by a large population of the NodeJs community and could now be considered as stable.
Well, this isn’t the best news. The really awesome news is that I have now some time to start the development of the v0.2 branch. This paves the way to some exciting developments which I am going to present in more details.
Backward compatibility
We say in french that we can’t prepare scramble eggs without breaking some eggs. So yes, the API will change, for the best. Since the beginning of the project, it felt wrong and the incomplete support of the stream API makes it non standard at best and, to be honest, hard to understand and use properly.
If your project rely on the 0.0.x or 0.1.x versions, don’t feel nervous. We will maintain support and any bug that we found will be ported to those versions.
The new Stream API
This little schema illustrates the structure of the future code:
|------------| |------------|------------| |------------|
| | | | | | |
| csv.from | | CSV | | csv.to |
| | | | | | |
| Stream | | Writer | Reader | | Stream |
| Reader | .pipe( | API | API | ).pipe( | Writer | )
| | | | | | |
| | | | | | |
|------------| |------------|------------| |------------|A new CSV instance will be fully compliant with the stream API. It will be both an Stream Writer to send input data and a Stream Reader to access output data.
Example:
fs.createReadStream( './in' )
.pipe( csv() )
.pipe( fs.createWriteStream('./out') )Convenient functionalities
Alternatively, it will come with convenient functions accessible by the from and to properties. Some of those functions were already present in the 0.1 release and are simply renamed. For exemple, the csv.fromPath() function is now csv.from.path(). New functions are planned such a the csv.to.string and the csv.to.array.
Example:
csv()
.from.path( './in' )
.to.path( './out' )Code isolation
Until now, all the source code present inside a single file. From now on, the code layout will be:
- ”./lib/from.js” Input convenient functions
- ”./lib/to.js” Output convenient functions
- ”./lib/csv.js” Core function implement the Stream API
- ”./lib/options.js” Default input and output options
- ”./lib/state.js” Default state object
This refactoring shall simplify the understanding of the source code and help new comers to prepare pull request which are particularly welcome.
Documentation
Like I have done in the past in many projects like Mecano (now Nikita), the readme content will be reduced to a minimum and the documentation will be generated directly from the source code. A small script will be written specifically from that purpose. The idea is to document each function with comments written in a markdown syntax. A simple regexp parser read each files, extract the comment and write markdown file inside a “./doc” folder. The doc folder is finally copied into the Jekyll directory of this website before being uploaded.
I find it much easier to document a project on a per function basis than by maintaining coherent a doc forlder or a large readme file.
CoffeeScript
This project was probably the last one I could claim to be written in JavaScript. I’m not here to start a war. I didn’t choose CoffeeScript to be cool, I use it for a year and a half now, and I find writting old JS to be a pain. The test have be translated some time ago but the production code was still in JavaScript. Starting from version 0.2.0, the code will be maitain in CoffeeScript. The source file will be located inside the “./src” folder and the transpiled JavaScript will be located inside the “./lib” folder.
Conclusion
It is an excellent time to watch the development of this new version and come with your own feedback. I’ll be happy to listen to your suggestions.
